ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Advanced Bayesian Machine Learning
Chapter 1: Revisiting Bayesian Foundations
Probabilistic Modeling Principles
Bayes' Theorem in High Dimensions
Subjectivity and Objectivity in Prior Selection
Information Theory Connection: Entropy and KL Divergence
Computational Challenges in Bayesian Inference
Advanced Model Checking and Criticism
Chapter 2: Markov Chain Monte Carlo Methods
Principles of Monte Carlo Integration
Markov Chain Theory for MCMC
Metropolis-Hastings Algorithm Variants
Gibbs Sampling for Conditional Structures
Hamiltonian Monte Carlo (HMC) Mechanics
The No-U-Turn Sampler (NUTS)
Diagnosing MCMC Convergence and Performance
Hands-on Practical: Implementing Advanced MCMC
Chapter 3: Variational Inference Techniques
Optimization as Inference: The VI Perspective
Deriving the Evidence Lower Bound (ELBO)
Mean-Field Approximation Details
Coordinate Ascent Variational Inference (CAVI)
Stochastic Variational Inference (SVI) for Large Data
Black Box Variational Inference (BBVI)
Advanced Variational Families
Comparing MCMC and VI Strengths
Hands-on Practical: Scalable Variational Inference
Chapter 4: Gaussian Processes
Bayesian Non-parametric Modeling Introduction
Defining Gaussian Processes: Priors Over Functions
Covariance Functions (Kernels): Properties and Selection
Gaussian Process Regression Formulation
Hyperparameter Marginal Likelihood Optimization
Gaussian Process Classification Techniques
Approximations for Scalable Gaussian Processes
Hands-on Practical: Gaussian Process Modeling
Chapter 5: Advanced Topics in Probabilistic Graphical Models
Bayesian Networks: Structure Learning
Parameter Learning in Bayesian Networks
Advanced Inference in PGMs: Junction Tree Algorithm
Approximate Inference in Large PGMs
Latent Dirichlet Allocation (LDA): Bayesian Formulation
Inference for LDA: Collapsed Gibbs Sampling
Inference for LDA: Variational Bayes
Hands-on Practical: Topic Modeling with LDA
Chapter 6: Bayesian Deep Learning
Motivation for Bayesian Deep Learning
Bayesian Neural Networks (BNNs): Priors over Weights
Inference Challenges in BNNs
MCMC Methods for BNNs (e.g., Stochastic Gradient HMC)
Variational Inference for BNNs (e.g., Bayes by Backprop)
Uncertainty Estimation in BNNs
Variational Autoencoders (VAEs) as Probabilistic Models
Dropout as Approximate Bayesian Inference
Practical Training and Evaluation of BNNs
Hands-on Practical: Building a Bayesian Neural Network

Coordinate Ascent Variational Inference (CAVI)

Coordinate Ascent Variational Inference (CAVI) is a technique used to find the best approximation q(z)q(\mathbf{z})q(z) to the true posterior p(z∣x)p(\mathbf{z}|\mathbf{x})p(z∣x) within a chosen family Q\mathcal{Q}Q. This is typically achieved by maximizing the Evidence Lower Bound (ELBO). Variational Inference (VI) broadly operates on this principle. A common choice for the approximation is the mean-field variational family, which assumes independence between latent variables (or groups of variables) as: q(z)=∏j=1Mqj(zj)q(\mathbf{z}) = \prod_{j=1}^M q_j(z_j)q(z)=∏j=1M​qj​(zj​) Here, z={z1,…,zM}\mathbf{z} = \{z_1, \dots, z_M\}z={z1​,…,zM​} are the latent variables. The primary challenge then becomes: how to find the optimal factors qj(zj)q_j(z_j)qj​(zj​) that maximize the ELBO, L(q)\mathcal{L}(q)L(q)?

CAVI provides an iterative approach. It maximizes the ELBO with respect to each factor qj(zj)q_j(z_j)qj​(zj​) one at a time, holding all other factors qi≠j(zi)q_{i \neq j}(z_i)qi=j​(zi​) fixed. This is analogous to coordinate ascent optimization applied in the functional space of distributions.

Deriving the CAVI Update Rule

Let's focus on optimizing the ELBO with respect to a single factor, say qj(zj)q_j(z_j)qj​(zj​). Recall the ELBO: L(q)=Eq[log⁡p(x,z)]−Eq[log⁡q(z)]\mathcal{L}(q) = \mathbb{E}_{q} [\log p(\mathbf{x}, \mathbf{z})] - \mathbb{E}_{q} [\log q(\mathbf{z})]L(q)=Eq​[logp(x,z)]−Eq​[logq(z)] Substituting the mean-field assumption q(z)=∏i=1Mqi(zi)q(\mathbf{z}) = \prod_{i=1}^M q_i(z_i)q(z)=∏i=1M​qi​(zi​): L(q)=∫ ⁣⋯∫(∏i=1Mqi(zi))(log⁡p(x,z)−∑i=1Mlog⁡qi(zi))dz1…dzM\mathcal{L}(q) = \int \dots \int \left( \prod_{i=1}^M q_i(z_i) \right) \left( \log p(\mathbf{x}, \mathbf{z}) - \sum_{i=1}^M \log q_i(z_i) \right) dz_1 \dots dz_ML(q)=∫⋯∫(∏i=1M​qi​(zi​))(logp(x,z)−∑i=1M​logqi​(zi​))dz1​…dzM​ We want to find the qj(zj)q_j(z_j)qj​(zj​) that maximizes this expression, holding qi≠jq_{i \neq j}qi=j​ fixed. Let's isolate the terms involving qj(zj)q_j(z_j)qj​(zj​): L(q)=∫qj(zj)(∫ ⁣⋯∫(∏i≠jqi(zi))log⁡p(x,z)(∏i≠jdzi))dzj−∫qj(zj)log⁡qj(zj)dzj+const\mathcal{L}(q) = \int q_j(z_j) \left( \int \dots \int \left( \prod_{i \neq j} q_i(z_i) \right) \log p(\mathbf{x}, \mathbf{z}) \left( \prod_{i \neq j} dz_i \right) \right) dz_j - \int q_j(z_j) \log q_j(z_j) dz_j + \text{const}L(q)=∫qj​(zj​)(∫⋯∫(∏i=j​qi​(zi​))logp(x,z)(∏i=j​dzi​))dzj​−∫qj​(zj​)logqj​(zj​)dzj​+const Here, "const" represents terms that do not depend on qj(zj)q_j(z_j)qj​(zj​). The term inside the large parentheses in the first integral is the expectation of the log joint distribution with respect to all factors except qjq_jqj​. Let's denote this as Ei≠j[log⁡p(x,z)]\mathbb{E}_{i \neq j} [\log p(\mathbf{x}, \mathbf{z})]Ei=j​[logp(x,z)]. So we have: L(q)=∫qj(zj)Ei≠j[log⁡p(x,z)]dzj−∫qj(zj)log⁡qj(zj)dzj+const\mathcal{L}(q) = \int q_j(z_j) \mathbb{E}_{i \neq j} [\log p(\mathbf{x}, \mathbf{z})] dz_j - \int q_j(z_j) \log q_j(z_j) dz_j + \text{const}L(q)=∫qj​(zj​)Ei=j​[logp(x,z)]dzj​−∫qj​(zj​)logqj​(zj​)dzj​+const Let log⁡p~j(zj)=Ei≠j[log⁡p(x,z)]\log \tilde{p}_j(z_j) = \mathbb{E}_{i \neq j} [\log p(\mathbf{x}, \mathbf{z})]logp~​j​(zj​)=Ei=j​[logp(x,z)]. The expression becomes: L(q)=∫qj(zj)log⁡p~j(zj)dzj−∫qj(zj)log⁡qj(zj)dzj+const\mathcal{L}(q) = \int q_j(z_j) \log \tilde{p}_j(z_j) dz_j - \int q_j(z_j) \log q_j(z_j) dz_j + \text{const}L(q)=∫qj​(zj​)logp~​j​(zj​)dzj​−∫qj​(zj​)logqj​(zj​)dzj​+const L(q)=−∫qj(zj)log⁡qj(zj)p~j(zj)dzj+const\mathcal{L}(q) = - \int q_j(z_j) \log \frac{q_j(z_j)}{\tilde{p}_j(z_j)} dz_j + \text{const}L(q)=−∫qj​(zj​)logp~​j​(zj​)qj​(zj​)​dzj​+const L(q)=−KL(qj(zj)∣∣p~j(zj))+const\mathcal{L}(q) = -KL(q_j(z_j) || \tilde{p}_j(z_j)) + \text{const}L(q)=−KL(qj​(zj​)∣∣p~​j​(zj​))+const Maximizing L(q)\mathcal{L}(q)L(q) with respect to qj(zj)q_j(z_j)qj​(zj​) is equivalent to minimizing the Kullback-Leibler (KL) divergence between qj(zj)q_j(z_j)qj​(zj​) and p~j(zj)\tilde{p}_j(z_j)p~​j​(zj​). The KL divergence is minimized (becomes zero) when qj(zj)=p~j(zj)q_j(z_j) = \tilde{p}_j(z_j)qj​(zj​)=p~​j​(zj​). Therefore, the optimal solution qj∗(zj)q_j^*(z_j)qj∗​(zj​) satisfies: log⁡qj∗(zj)=Ei≠j[log⁡p(x,z)]+const\log q_j^*(z_j) = \mathbb{E}_{i \neq j} [\log p(\mathbf{x}, \mathbf{z})] + \text{const}logqj∗​(zj​)=Ei=j​[logp(x,z)]+const Exponentiating both sides gives: qj∗(zj)∝exp⁡(Ei≠j[log⁡p(x,z)])q_j^*(z_j) \propto \exp \left( \mathbb{E}_{i \neq j} [\log p(\mathbf{x}, \mathbf{z})] \right)qj∗​(zj​)∝exp(Ei=j​[logp(x,z)]) The constant term is determined by the normalization requirement ∫qj∗(zj)dzj=1\int q_j^*(z_j) dz_j = 1∫qj∗​(zj​)dzj​=1.

This is the core update rule for CAVI. It states that the optimal distribution for the jjj-th factor qj(zj)q_j(z_j)qj​(zj​) is proportional to the exponentiated expected log of the joint probability p(x,z)p(\mathbf{x}, \mathbf{z})p(x,z), where the expectation is taken with respect to the current distributions of all other factors qi≠j(zi)q_{i \neq j}(z_i)qi=j​(zi​).

The CAVI Algorithm

The CAVI algorithm proceeds as follows:

  1. Initialization: Initialize the parameters of each variational factor qj(zj)q_j(z_j)qj​(zj​) for j=1,…,Mj=1, \dots, Mj=1,…,M. This might involve initializing means, variances, or other relevant parameters depending on the distributional form chosen for each qjq_jqj​.
  2. Iteration: Repeat until convergence:
    • For each factor j=1,…,Mj = 1, \dots, Mj=1,…,M:
      • Compute the expectation Ei≠j[log⁡p(x,z)]\mathbb{E}_{i \neq j} [\log p(\mathbf{x}, \mathbf{z})]Ei=j​[logp(x,z)]. This requires using the current parameters of the distributions qi≠j(zi)q_{i \neq j}(z_i)qi=j​(zi​).
      • Update the factor qj(zj)q_j(z_j)qj​(zj​) using the rule: qj(zj)←exp⁡(Ei≠j[log⁡p(x,z)])∫exp⁡(Ei≠j[log⁡p(x,z)])dzjq_j(z_j) \leftarrow \frac{\exp \left( \mathbb{E}_{i \neq j} [\log p(\mathbf{x}, \mathbf{z})] \right)}{\int \exp \left( \mathbb{E}_{i \neq j} [\log p(\mathbf{x}, \mathbf{z})] \right) dz_j}qj​(zj​)←∫exp(Ei=j​[logp(x,z)])dzj​exp(Ei=j​[logp(x,z)])​ In practice, this often means updating the parameters of the distribution qj(zj)q_j(z_j)qj​(zj​) based on the computed expectation.
  3. Convergence Check: Monitor the ELBO or the parameters of the variational factors. Stop when the change between iterations falls below a predefined threshold.

The CAVI algorithm iteratively updates each factor qj(zj)q_j(z_j)qj​(zj​) based on the expectation of the log joint probability, conditioned on the current state of the other factors, until convergence.

The Role of Conjugacy

The practicality of CAVI often relies on the structure of the model p(x,z)p(\mathbf{x}, \mathbf{z})p(x,z). If the complete conditional p(zj∣x,z¬j)p(z_j | \mathbf{x}, \mathbf{z}_{\neg j})p(zj​∣x,z¬j​) (where z¬j\mathbf{z}_{\neg j}z¬j​ denotes all variables except zjz_jzj​) is in the same family as the prior for zjz_jzj​, the model exhibits conditional conjugacy. In such cases, the CAVI update for qj∗(zj)q_j^*(z_j)qj∗​(zj​) often results in a distribution of the same parametric form as the initial choice for qj(zj)q_j(z_j)qj​(zj​).

For instance, if qj(zj)q_j(z_j)qj​(zj​) is chosen to be Gaussian, and the model structure allows, the update qj∗(zj)q_j^*(z_j)qj∗​(zj​) derived from the expectation Ei≠j[log⁡p(x,z)]\mathbb{E}_{i \neq j} [\log p(\mathbf{x}, \mathbf{z})]Ei=j​[logp(x,z)] might also be Gaussian. The update step then simplifies to calculating the new mean and variance parameters based on the expected values (moments) derived from the other factors qi≠jq_{i \neq j}qi=j​. This makes the implementation significantly easier, as we only need to track and update the parameters (e.g., means, variances) of the factors.

Advantages and Disadvantages of CAVI

Advantages:

  • Guaranteed Convergence: The ELBO is guaranteed to increase (or stay the same) at each step of CAVI, ensuring convergence to a local optimum.
  • Deterministic: Unlike MCMC, CAVI is a deterministic optimization algorithm, yielding the same result given the same initialization.
  • Simplicity (with Conjugacy): When conditional conjugacy applies, the updates can often be derived analytically and involve updating parameters of standard distributions.

Disadvantages:

  • Local Optima: CAVI optimizes the ELBO, which is generally non-convex. The algorithm may converge to a local optimum, potentially providing a poor approximation of the true posterior. Different initializations might lead to different results.
  • Mean-Field Limitation: The accuracy is fundamentally limited by the expressiveness of the chosen variational family. The mean-field assumption (independence between factors) might be too restrictive for models with strong posterior dependencies.
  • Analytical Calculations: Deriving the update rule requires calculating Ei≠j[log⁡p(x,z)]\mathbb{E}_{i \neq j} [\log p(\mathbf{x}, \mathbf{z})]Ei=j​[logp(x,z)]. This can be complex or analytically intractable for some models.
  • Scalability: Traditional CAVI requires iterating through the entire dataset to compute expectations for global variable updates (parameters governing all data points), which doesn't scale well to very large datasets.

CAVI provides a foundational understanding of variational optimization. While effective for moderately sized problems with suitable model structures, its limitations, particularly regarding scalability and the need for analytical derivations, motivate the development of more advanced techniques like Stochastic Variational Inference (SVI) and Black Box Variational Inference (BBVI), which we will explore next.

  • Pattern Recognition and Machine Learning, Christopher M. Bishop, 2006 (Springer) DOI: 10.1007/978-0-387-45528-0 - Provides a comprehensive and foundational treatment of Variational Inference, including the derivation and algorithm of Coordinate Ascent Variational Inference (CAVI) within the mean-field framework.
  • Variational Inference: A Review for Statisticians, David M. Blei, Alp Kucukelbir, Jon D. McAuliffe, 2017 Journal of the American Statistical Association, Vol. 112 (Taylor & Francis) DOI: 10.1080/01621459.2017.1328493 - An authoritative review paper that provides a broad overview of variational inference, detailing the theoretical underpinnings and practical applications of CAVI.
  • Probabilistic Machine Learning: Advanced Topics, Kevin Patrick Murphy, 2023 (MIT Press) - Offers an in-depth and contemporary treatment of variational inference, with detailed discussions on the mechanics and application of CAVI.
  • Latent Dirichlet Allocation, David M. Blei, Andrew Y. Ng, and Michael I. Jordan, 2003 Journal of Machine Learning Research, Vol. 3 - This seminal paper applies the mean-field coordinate ascent approach to a probabilistic topic model (Latent Dirichlet Allocation), illustrating the practical implementation and effectiveness of CAVI for complex generative models.

© 2026 ApX Machine Learning