Introduction to Diffusion Model Concepts

While Generative Adversarial Networks (GANs) represent one major family of generative models, characterized by their adversarial training dynamic, Diffusion Models offer a distinct and increasingly powerful approach. Introduced more recently, these models have rapidly achieved state-of-the-art results, particularly in image synthesis, often producing high-fidelity samples with remarkable diversity. Unlike the sometimes unstable min-max game of GANs, diffusion models typically offer a more stable training process, although often at the cost of slower sampling speeds.

The fundamental idea behind diffusion models involves two processes: a fixed forward process and a learned reverse process.

The Forward Process: Gradually Adding Noise

Imagine taking a clean data sample, like an image $x_0$ drawn from the true data distribution $p_{data}(x)$. The forward process systematically degrades this sample over a series of $T$ timesteps by gradually adding small amounts of Gaussian noise. This defines a Markov chain where the state at timestep $t$, denoted $x_t$, depends only on the state at the previous timestep $x_{t-1}$:

$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t \mathbf{I})$

Here, $\beta_t$ represents a small positive constant determining the variance of the noise added at step $t$. The sequence $\beta_1, \beta_2, ..., \beta_T$ is known as the variance schedule, which is typically predefined (e.g., increasing linearly or quadratically).

A useful property of this process is that we can directly sample $x_t$ conditioned on the original data $x_0$ without iterating through all intermediate steps. If we define $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{i=1}^t \alpha_i$, then the distribution $q(x_t | x_0)$ is also Gaussian:

$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1 - \bar{\alpha}_t) \mathbf{I})$

As $t$ increases towards $T$, the cumulative effect of adding noise means $\bar{\alpha}_t$ approaches zero. Consequently, $x_T$ loses almost all information about the original $x_0$ and effectively becomes indistinguishable from pure Gaussian noise, $x_T \approx \mathcal{N}(0, \mathbf{I})$. The forward process is fixed; it requires no training.

Diagram illustrating the fixed forward process, where data $x_0$ is gradually noised over T steps according to the transition probability $q$ until it resembles pure noise $x_T$.

The Reverse Process: Learning to Denoise

The core generative task lies in the reverse process. We want to start with random noise $x_T \sim \mathcal{N}(0, \mathbf{I})$ and reverse the noising steps to gradually denoise it, eventually obtaining a sample $x_0$ that looks like it came from the original data distribution $p_{data}$. This involves learning the transition probabilities $p_\theta(x_{t-1} | x_t)$ for $t = T, T-1, ..., 1$.

If the noise steps $\beta_t$ in the forward process are sufficiently small, the true reverse transition $q(x_{t-1} | x_t, x_0)$ can also be shown to be approximately Gaussian. However, calculating this true reverse transition requires knowing the original data $x_0$, which is precisely what we want to generate.

Therefore, we approximate these true reverse transitions using a neural network, parameterized by $\theta$. This network takes the noisy sample $x_t$ and the current timestep $t$ as input and learns to predict the parameters (typically the mean and variance) of the distribution $p_\theta(x_{t-1} | x_t)$:

$p_\theta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))$

Instead of directly predicting the mean $\mu_\theta(x_t, t)$, a common and effective parameterization is to train the network, often denoted $\epsilon_\theta(x_t, t)$, to predict the noise $\epsilon$ that was added to get from $x_{t-1}$ to $x_t$ (or more accurately, the noise component corresponding to $x_t$ based on $x_0$). Given the predicted noise $\epsilon_\theta(x_t, t)$, we can derive the parameters for $p_\theta(x_{t-1} | x_t)$.

Training and Generation

The parameters $\theta$ of the reverse process network are optimized by maximizing the likelihood of the training data. This often involves optimizing a variational lower bound (ELBO) on the log-likelihood. For diffusion models, this objective can often be simplified to a much more tractable form. A common simplified objective aims to minimize the mean squared error between the actual noise $\epsilon$ added during the forward process (which can be easily sampled given $x_0$) and the noise predicted by the network $\epsilon_\theta$:

$L_{simple}(\theta) = \mathbb{E}_{t \sim [1, T], x_0 \sim p_{data}, \epsilon \sim \mathcal{N}(0, \mathbf{I})} \left[ || \epsilon - \epsilon_\theta(x_t, t) ||^2 \right]$

where $x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\epsilon$ is generated using the direct forward sampling formula. Training involves repeatedly sampling a data point $x_0$, a timestep $t$, sampling noise $\epsilon$, constructing the corresponding $x_t$, and performing a gradient descent step on this loss.

Once the model $\epsilon_\theta(x_t, t)$ is trained, generation (sampling) proceeds by:

1. Sampling initial noise $x_T \sim \mathcal{N}(0, \mathbf{I})$.
2. Iterating backwards from $t = T$ down to $t = 1$:
   • Use the network $\epsilon_\theta(x_t, t)$ to predict the noise component.
   • Use this prediction to calculate the parameters (mean $\mu_\theta(x_t, t)$ and variance $\Sigma_\theta(x_t, t)$) of the distribution $p_\theta(x_{t-1} | x_t)$.
   • Sample $x_{t-1} \sim p_\theta(x_{t-1} | x_t)$.
3. The final output $x_0$ is the generated sample.

Diagram comparing the fixed forward noising process $q$ with the learned reverse denoising process $p_\theta$. The reverse process starts from noise $x_T$ and uses a trained model $\epsilon_\theta$ (optimized via the training objective $L_{simple}$) to iteratively sample less noisy states, ultimately producing a generated sample $x_0$.

This iterative noise-and-denoise framework forms the basis of diffusion models. While straightforward, the specific choices of variance schedules, network architectures (often U-Nets), and training objectives lead to different model variants like Denoising Diffusion Probabilistic Models (DDPMs) and score-based models, which we will explore in detail in Chapter 4. Understanding this core mechanism is essential before implementing and optimizing these advanced generative techniques.