Understanding Sampling Variance

In the previous section, we outlined the general procedure for generating data using the reverse diffusion process, starting from noise $x_T$ and iteratively applying the learned denoising step to arrive at $x_0$. A significant characteristic of the standard Denoising Diffusion Probabilistic Models (DDPM) sampling process is its inherent stochasticity. Even if you were to start two generation processes from the exact same initial noise tensor $x_T$, you would likely end up with two different final samples $x_0$. Let's examine why this happens.

Recall from Chapter 3 that the reverse process aims to approximate the true posterior $p(x_{t-1}|x_t)$. In DDPM, we parameterize this reverse transition at each step $t$ as a Gaussian distribution:

$p_\theta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \sigma_t^2 \mathbf{I})$

Here, $\mu_\theta(x_t, t)$ represents the mean of the distribution for the previous state $x_{t-1}$, given the current state $x_t$ and timestep $t$. Our trained neural network $\epsilon_\theta(x_t, t)$ is used to calculate this mean. The term $\sigma_t^2 \mathbf{I}$ represents the variance, where $\sigma_t^2$ is typically a hyperparameter derived from the forward process noise schedule (often related to $\beta_t$ or $\tilde{\beta}_t$).

The important point is that obtaining $x_{t-1}$ from $x_t$ involves sampling from this Gaussian distribution, not just calculating the mean $\mu_\theta(x_t, t)$. The sampling operation looks like this:

$x_{t-1} = \mu_\theta(x_t, t) + \sigma_t z$

where $z$ is a random vector sampled from a standard Gaussian distribution, $z \sim \mathcal{N}(0, \mathbf{I})$.

This addition of the scaled random noise $\sigma_t z$ at every step $t$ (from $T$ down to 1) is the source of the variability in the DDPM sampling process. Each step introduces a small amount of randomness.

The Cumulative Effect of Step-wise Randomness

Imagine the path from pure noise $x_T$ to the final sample $x_0$ as a sequence of $T$ steps. At each step, the model predicts the direction (the mean $\mu_\theta$), but then takes a slightly randomized step in that direction due to the added noise $\sigma_t z$.

This diagram shows how, starting from the same state $x_t$, sampling different noise vectors $z_t$ and $z'_t$ during the reverse step leads to slightly different states $x_{t-1}$ for two potential generation paths. Over many steps, these small divergences accumulate, resulting in distinct final samples $x_0$.

Because these small random perturbations accumulate over the entire sequence of $T$ steps, the final output $x_0$ reflects the sum of these random choices. This explains why running the DDPM sampling process multiple times, even with the same hyperparameters and trained model, yields a diverse set of generated samples. This inherent randomness is often desirable, as it allows a single trained model to generate a wide variety of outputs.

The Role of Variance Schedule ($\sigma_t^2$)

The magnitude of the variance $\sigma_t^2$ in the reverse step influences how much randomness is injected at each step. In the original DDPM paper, $\sigma_t^2$ is set based on the noise schedule $\beta_t$ used in the forward process (specifically, $\sigma_t^2 = \tilde{\beta}_t = \frac{1 - \bar{\alpha}_{t-1}}{1 - \bar{\alpha}_t} \beta_t$ or simply $\sigma_t^2 = \beta_t$). While this is a standard choice tied theoretically to the forward process, it is possible to use different values. Larger values of $\sigma_t^2$ generally lead to more diversity in the generated samples but might slightly reduce the quality or faithfulness to the learned data distribution if set too high. Smaller values reduce the stochasticity.

Connection to Faster Sampling Methods

Understanding this source of variance is also important when we consider faster sampling methods like Denoising Diffusion Implicit Models (DDIM), which we will discuss next. DDIM reinterprets the generation process and introduces a parameter (often denoted $\eta$) that controls the amount of stochasticity. When $\eta=0$, DDIM sampling becomes deterministic given a starting noise $x_T$. This means for a fixed $x_T$, DDIM with $\eta=0$ will always produce the same $x_0$. This contrasts sharply with DDPM, which is inherently stochastic due to the $\sigma_t z$ term added at each step. This trade-off between stochasticity (diversity) and determinism (reproducibility, potentially faster sampling) is a primary difference between DDPM and DDIM sampling strategies.