Improved Techniques: DDIM and Variance Schedules

While Denoising Diffusion Probabilistic Models (DDPMs) achieve impressive generation quality, a significant drawback is their slow sampling speed. Generating a single sample often requires hundreds or thousands of sequential denoising steps, corresponding to the number of noise levels $T$ used during training. Techniques designed to accelerate sampling and refine the generation process are examined, primarily focusing on Denoising Diffusion Implicit Models (DDIM) and the impact of variance schedules.

Denoising Diffusion Implicit Models (DDIM) for Faster Sampling

DDIM offers a modification to the generative (reverse) process of DDPMs that allows for much faster sampling, often reducing the number of required steps by 10-100x without needing to retrain the model. The innovation lies in formulating a non-Markovian reverse process that still uses the same noise prediction network $\boldsymbol{\epsilon}_\theta$ trained with the DDPM objective.

Recall the standard DDPM reverse step:

$$p_\theta(\mathbf{x}_{t-1} | \mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \tilde{\beta}_t \mathbf{I})$$

where $\boldsymbol{\mu}_\theta$ depends on $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$ and the variance $\tilde{\beta}_t$ is fixed based on the noise schedule $\beta_t$. This process is Markovian, meaning $\mathbf{x}_{t-1}$ depends only on $\mathbf{x}_t$.

DDIM introduces a more general family of non-Markovian diffusion processes. The core idea involves first predicting the final clean data point $\mathbf{x}_0$ from the current noisy state $\mathbf{x}_t$, and then using this prediction to guide the step towards $\mathbf{x}_{t-1}$. The predicted $\mathbf{x}_0$ is obtained by rearranging the forward process equation $\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon}$:

$$\mathbf{x}_0^{\text{pred}}(t) = \frac{1}{\sqrt{\bar{\alpha}_t}}(\mathbf{x}_t - \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t))$$

This predicted $\mathbf{x}_0$ represents the model's best estimate of the original data given the noisy input $\mathbf{x}_t$ and the current time step $t$.

The DDIM reverse step then samples $\mathbf{x}_{t-1}$ using this predicted $\mathbf{x}_0$:

$$\mathbf{x}_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\mathbf{x}_0^{\text{pred}}(t) + \underbrace{\sqrt{1 - \bar{\alpha}_{t-1} - \sigma_t^2}}_{\text{direction pointing to } \mathbf{x}_t} \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t) + \underbrace{\sigma_t \boldsymbol{\epsilon}'}_{\text{random noise}}$$

Here, $\boldsymbol{\epsilon}' \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ is fresh random noise, and $\sigma_t$ controls the stochasticity of this reverse step. The parameter $\sigma_t$ is defined using a hyperparameter $\eta \ge 0$:

$$\sigma_t(\eta) = \eta \sqrt{\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t}} \sqrt{1 - \frac{\bar{\alpha}_t}{\bar{\alpha}_{t-1}}}$$

The insight is the role of $\eta$:

• Stochastic Case ($\eta = 1$): When $\eta = 1$, the value of $\sigma_t^2$ becomes equal to the DDPM variance $\tilde{\beta}_t = \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t} \beta_t$. In this specific case, the DDIM sampling process recovers the original DDPM Markovian process.
• Deterministic Case ($\eta = 0$): When $\eta = 0$, we have $\sigma_t = 0$. The random noise term vanishes, and the update becomes fully deterministic given $\mathbf{x}_t$: $$\mathbf{x}_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\mathbf{x}_0^{\text{pred}}(t) + \sqrt{1 - \bar{\alpha}_{t-1}} \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$$ This makes the generative process implicit because $\mathbf{x}_{t-1}$ is computed directly, not sampled from a distribution. This deterministic nature allows for significantly larger jumps in the time steps during sampling. Instead of using all $T$ steps (e.g., $T=1000$), we can use a subsequence of time steps $\tau_1, \tau_2, ..., \tau_S$ where $S \ll T$ (e.g., $S=50$ or $S=100$). The update rule is applied successively for $t = \tau_S, \tau_{S-1}, ..., \tau_1$. This deterministic variant is often associated with the Probability Flow Ordinary Differential Equation (ODE) formulation of diffusion models.

Comparison between the DDPM reverse step and the deterministic DDIM reverse step ($\eta=0$). DDIM uses an intermediate prediction of the clean data $\mathbf{x}_0$ to determine $\mathbf{x}_{t-1}$.

Using $\eta=0$ (deterministic DDIM) typically yields high-quality samples with far fewer steps. Values of $\eta$ between 0 and 1 allow interpolating between deterministic and stochastic generation, potentially adding diversity at the cost of some consistency. A major advantage of DDIM is that it uses the exact same network $\boldsymbol{\epsilon}_\theta$ trained for DDPMs. Only the sampling procedure changes, making it easy to deploy for faster generation with existing models.

Variance Schedules

The choice of the noise schedule, defined by $\beta_t$ for $t=1, ..., T$, is another important aspect influencing model performance. This schedule determines how quickly noise is added in the forward process, controlling the signal-to-noise ratio at each step $t$. Common schedules include:

1. Linear Schedule: $\beta_t$ increases linearly from a small value $\beta_1$ (e.g., $10^{-4}$) to a larger value $\beta_T$ (e.g., $0.02$). This was used in the original DDPM paper.
2. Cosine Schedule: Proposed to improve training stability and sample quality. The cumulative noise level $\bar{\alpha}_t$ follows a cosine shape, preventing the signal from decaying too quickly early in the forward process. Specifically: $$\bar{\alpha}_t = \frac{f(t)}{f(0)}, \quad \text{where} \quad f(t) = \cos\left(\frac{t/T + s}{1+s} \cdot \frac{\pi}{2}\right)^2$$ Here, $s$ is a small offset (e.g., $0.008$) to prevent $\beta_t$ from being too small near $t=0$. $\beta_t$ is then derived as $\beta_t = 1 - \frac{\bar{\alpha}_t}{\bar{\alpha}_{t-1}}$.

The square root of $\bar{\alpha}_t$, representing the signal rate, decreases over time. A linear schedule for $\beta_t$ results in a roughly linear decrease in $\sqrt{\bar{\alpha}_t}$, while a cosine schedule maintains a higher signal rate for longer before decaying more rapidly.

Beyond fixed schedules, some research has explored learning the variance of the reverse process $p_\theta(\mathbf{x}_{t-1} | \mathbf{x}_t)$. The original DDPM fixes this variance to $\tilde{\beta}_t \mathbf{I}$ or $\beta_t \mathbf{I}$. However, the model $\boldsymbol{\epsilon}_\theta$ can be modified to also predict a parameter $v$ that interpolates between these lower and upper bounds on the optimal reverse variance. While learning the variance can improve log-likelihood scores, it often doesn't significantly enhance perceptual quality (measured by metrics like FID) and adds complexity. The fixed, small variance approach (often approximated by $\tilde{\beta}_t$) generally works well in practice. The DDIM framework sidesteps explicit variance learning by controlling stochasticity via $\eta$, offering a flexible way to manage the reverse process variance implicitly.

"In summary, DDIM provides a powerful method for accelerating diffusion model sampling by defining a deterministic or near-deterministic reverse path, leveraging the same trained noise prediction network. The choice of variance schedule ($\beta_t$) remains an important design decision affecting model performance, with cosine schedules often being preferred over linear ones. These techniques collectively make diffusion models more practical for applications requiring efficient generation."