Gibbs Sampling for Conditional Structures

While Metropolis-Hastings offers a general framework for drawing samples from tricky posterior distributions, it relies on finding a good proposal distribution, $q(\theta'|\theta^{(t-1)})$, which can be challenging. Tuning this proposal is often more art than science. Gibbs sampling provides an alternative MCMC approach that elegantly sidesteps the need for explicit proposal distributions, provided your model has a specific structure. It's particularly effective when the joint posterior distribution $p(\theta | D)$ is complex, but the full conditional distributions are easier to handle.

The Power of Conditionals

The core idea behind Gibbs sampling is simple: instead of trying to sample the entire parameter vector $\theta = (\theta_1, \theta_2, ..., \theta_k)$ simultaneously from the joint posterior $p(\theta | D)$, we sample each parameter (or block of parameters) individually, conditional on the current values of all other parameters.

The key requirement for Gibbs sampling is the ability to derive and sample from the full conditional distribution for each parameter $\theta_i$. This is the distribution of $\theta_i$ given all other parameters $\theta_{-i} = (\theta_1, ..., \theta_{i-1}, \theta_{i+1}, ..., \theta_k)$ and the data $D$:

$$p(\theta_i | \theta_{-i}, D) = p(\theta_i | \theta_1, ..., \theta_{i-1}, \theta_{i+1}, ..., \theta_k, D)$$

Why might this be easier? In many hierarchical models or models with conjugate priors, these full conditionals simplify considerably. Sometimes, they even turn out to be standard distributions (like Normal, Gamma, Beta, etc.) that we can easily sample from. This happens because conditioning on other variables effectively treats them as fixed constants within the expression for the conditional probability, often simplifying the functional form derived from the joint posterior. Remember that the joint posterior is proportional to the likelihood times the prior: $p(\theta|D) \propto p(D|\theta)p(\theta)$. The full conditional for $\theta_i$ is proportional to this joint density, viewed only as a function of $\theta_i$, holding all other $\theta_j$ ($j \neq i$) constant.

The Gibbs Sampling Algorithm

Let's say we want to draw samples from the joint posterior $p(\theta_1, ..., \theta_k | D)$. The Gibbs sampler proceeds as follows:

1. Initialization: Choose an initial set of parameter values $\theta^{(0)} = (\theta_1^{(0)}, \theta_2^{(0)}, ..., \theta_k^{(0)})$. This could be random, or based on prior knowledge or simpler estimation methods (like maximum likelihood estimates).
2. Iteration: For each iteration $t = 1, 2, ..., T$:
   • Sample $\theta_1^{(t)}$ from the full conditional: $$\theta_1^{(t)} \sim p(\theta_1 | \theta_2^{(t-1)}, \theta_3^{(t-1)}, ..., \theta_k^{(t-1)}, D)$$
   • Sample $\theta_2^{(t)}$ from the full conditional, using the most recently updated value for $\theta_1$: $$\theta_2^{(t)} \sim p(\theta_2 | \theta_1^{(t)}, \theta_3^{(t-1)}, ..., \theta_k^{(t-1)}, D)$$
   • Continue this process for all parameters up to $\theta_k$: $$\theta_i^{(t)} \sim p(\theta_i | \theta_1^{(t)}, ..., \theta_{i-1}^{(t)}, \theta_{i+1}^{(t-1)}, ..., \theta_k^{(t-1)}, D)$$
   • Finally, sample $\theta_k^{(t)}$: $$\theta_k^{(t)} \sim p(\theta_k | \theta_1^{(t)}, \theta_2^{(t)}, ..., \theta_{k-1}^{(t)}, D)$$
3. Output: The sequence of sampled vectors $(\theta^{(1)}, \theta^{(2)}, ..., \theta^{(T)})$ forms a Markov chain whose stationary distribution is the target posterior $p(\theta | D)$. After discarding an initial burn-in period, these samples can be used to approximate the posterior.

Notice that each parameter is updated using the latest available values of the other parameters within the same iteration. This sequential updating is characteristic of Gibbs sampling.

Gibbs sampling for a two-parameter model, $\theta = (\theta_1, \theta_2)$. Each step involves sampling one parameter conditional on the current value of the other, effectively moving parallel to the axes.

Why Does Gibbs Work?

Gibbs sampling can be viewed as a special instance of the Metropolis-Hastings algorithm. For the step updating $\theta_i$, the proposal distribution is simply the full conditional $p(\theta_i | \theta_{-i}^{(current)}, D)$. It turns out that with this specific proposal, the Metropolis-Hastings acceptance probability is always 1. Thus, every proposed sample is accepted, making the algorithm conceptually simpler and computationally efficient if sampling from the conditionals is fast. As with other MCMC methods, the sequence of samples forms a Markov chain that, under mild conditions, converges to the target posterior distribution as its stationary distribution.

Advantages and Considerations

Advantages:

• No Tuning Required (Mostly): Unlike Metropolis-Hastings, you don't need to design or tune proposal distributions. The "proposal" is intrinsically defined by the model structure via the full conditionals.
• High Acceptance Rate: Samples are effectively accepted 100% of the time.
• Simplicity: If the full conditional distributions are known and easy to sample from (e.g., standard distributions), implementation is often straightforward.

Considerations:

• Deriving Conditionals: The main prerequisite is the ability to derive and implement samplers for all full conditional distributions $p(\theta_i | \theta_{-i}, D)$. This is not always possible or easy. If even one conditional is intractable, standard Gibbs cannot be used directly (though hybrid methods might exist).
• Correlation and Mixing: Gibbs sampling can be inefficient if parameters are highly correlated in the posterior distribution. Imagine sampling from a distribution concentrated along a thin diagonal line. Taking steps parallel to the axes (as Gibbs does) will result in very small movements along the diagonal, leading to high autocorrelation between successive samples and slow exploration (mixing) of the posterior space. The chain might take a very long time to traverse the relevant regions of the parameter space.
• Convergence: While guaranteed in theory, convergence can be slow in practice, especially with high correlations. Careful convergence diagnostics (covered later) are essential.

Blocked Gibbs Sampling

One strategy to mitigate slow mixing due to correlations is Blocked Gibbs Sampling. Instead of sampling each $\theta_i$ individually, we can group highly correlated parameters into "blocks" and sample them together from their joint conditional distribution, conditional on all parameters outside the block. For example, if $\theta_1$ and $\theta_2$ are highly correlated but relatively independent of $\theta_3$, we might sample $(\theta_1, \theta_2)$ jointly from $p(\theta_1, \theta_2 | \theta_3, D)$, and then sample $\theta_3$ from $p(\theta_3 | \theta_1, \theta_2, D)$. This requires being able to sample from the joint conditional of the block, which might be feasible in some cases and can significantly improve mixing.

When to Use Gibbs Sampling

Gibbs sampling is a valuable tool in the Bayesian practitioner's MCMC toolkit. It shines when:

1. The model structure allows for easy derivation of the full conditional distributions for all parameters.
2. These conditional distributions are standard distributions from which samples can be readily generated.
3. Posterior correlations between parameters are not excessively high, or blocked sampling can be effectively used.

It's frequently used in hierarchical models and specific structures like Latent Dirichlet Allocation (LDA) for topic modeling, which we will encounter later in the course. Even when not all conditionals are tractable, Gibbs steps can sometimes be combined with Metropolis-Hastings steps within a hybrid sampler. Understanding Gibbs sampling is therefore important not only as a standalone algorithm but also as a building block for more complex MCMC strategies.