Calculating the posterior distribution $p(\mathbf{z}|\mathbf{x})$ is often computationally challenging due to the intractable integral required for the evidence $p(\mathbf{x})$. While Markov Chain Monte Carlo methods, discussed previously, provide a way to sample from the posterior, they can be computationally intensive, especially for large datasets or complex models.

This chapter introduces Variational Inference (VI), an alternative family of techniques for approximating posterior distributions. Instead of sampling, VI reframes Bayesian inference as an optimization problem. The goal is to find a distribution $q(\mathbf{z})$ from a tractable family that is closest to the true posterior $p(\mathbf{z}|\mathbf{x})$, typically by maximizing a lower bound on the model evidence, known as the Evidence Lower Bound (ELBO): $\mathcal{L}(q) = \mathbb{E}_{q(\mathbf{z})} [\log p(\mathbf{x}, \mathbf{z})] - \mathbb{E}_{q(\mathbf{z})} [\log q(\mathbf{z})]$

We will start by formulating inference as optimization and deriving the ELBO. You will learn about the common mean-field approximation and the Coordinate Ascent Variational Inference (CAVI) algorithm. We will then cover techniques for scaling inference to large datasets using Stochastic Variational Inference (SVI) and handle models with complex gradients using Black Box Variational Inference (BBVI). We will also look at more expressive variational families and analyze the trade-offs between VI and MCMC approaches regarding speed, accuracy, and scalability. Practical implementation using common libraries will solidify these concepts.