While Generative Adversarial Networks (GANs) represent one major family of generative models, characterized by their adversarial training dynamic, Diffusion Models offer a distinct and increasingly powerful approach. Introduced more recently, these models have rapidly achieved state-of-the-art results, particularly in image synthesis, often producing high-fidelity samples with remarkable diversity. Unlike the sometimes unstable min-max game of GANs, diffusion models typically offer a more stable training process, although often at the cost of slower sampling speeds.

The fundamental idea behind diffusion models involves two processes: a fixed forward process and a learned reverse process.

The Forward Process: Gradually Adding Noise

Imagine taking a clean data sample, like an image $x_0$ drawn from the true data distribution $p_{data}(x)$ . The forward process systematically degrades this sample over a series of $T$ timesteps by gradually adding small amounts of Gaussian noise. This defines a Markov chain where the state at timestep $t$ , denoted $x_t$ , depends only on the state at the previous timestep $x_{t-1}$ :

$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t \mathbf{I})$

Here, $\beta_t$ represents a small positive constant determining the variance of the noise added at step $t$ . The sequence $\beta_1, \beta_2, ..., \beta_T$ is known as the variance schedule, which is typically predefined (e.g., increasing linearly or quadratically).

A useful property of this process is that we can directly sample $x_t$ conditioned on the original data $x_0$ without iterating through all intermediate steps. If we define $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{i=1}^t \alpha_i$ , then the distribution $q(x_t | x_0)$ is also Gaussian:

$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1 - \bar{\alpha}_t) \mathbf{I})$

As $t$ increases towards $T$ , the cumulative effect of adding noise means $\bar{\alpha}_t$ approaches zero. Consequently, $x_T$ loses almost all information about the original $x_0$ and effectively becomes indistinguishable from pure Gaussian noise, $x_T \approx \mathcal{N}(0, \mathbf{I})$ . The forward process is fixed; it requires no training.

Diagram illustrating the fixed forward process, where data $x_0$ is gradually noised over T steps according to the transition probability $q$ until it resembles pure noise $x_T$ .

The Reverse Process: Learning to Denoise

The core generative task lies in the reverse process. We want to start with random noise $x_T \sim \mathcal{N}(0, \mathbf{I})$ and reverse the noising steps to gradually denoise it, eventually obtaining a sample $x_0$ that looks like it came from the original data distribution $p_{data}$ . This involves learning the transition probabilities $p_\theta(x_{t-1} | x_t)$ for $t = T, T-1, ..., 1$ .

If the noise steps $\beta_t$ in the forward process are sufficiently small, the true reverse transition $q(x_{t-1} | x_t, x_0)$ can also be shown to be approximately Gaussian. However, calculating this true reverse transition requires knowing the original data $x_0$ , which is precisely what we want to generate.

Therefore, we approximate these true reverse transitions using a neural network, parameterized by $\theta$ . This network takes the noisy sample $x_t$ and the current timestep $t$ as input and learns to predict the parameters (typically the mean and variance) of the distribution $p_\theta(x_{t-1} | x_t)$ :

$p_\theta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))$

Instead of directly predicting the mean $\mu_\theta(x_t, t)$ , a common and effective parameterization is to train the network, often denoted $\epsilon_\theta(x_t, t)$ , to predict the noise $\epsilon$ that was added to get from $x_{t-1}$ to $x_t$ (or more accurately, the noise component corresponding to $x_t$ based on $x_0$ ). Given the predicted noise $\epsilon_\theta(x_t, t)$ , we can derive the parameters for $p_\theta(x_{t-1} | x_t)$ .

Training and Generation

The parameters $\theta$ of the reverse process network are optimized by maximizing the likelihood of the training data. This often involves optimizing a variational lower bound (ELBO) on the log-likelihood. For diffusion models, this objective can often be simplified to a much more tractable form. A common simplified objective aims to minimize the mean squared error between the actual noise $\epsilon$ added during the forward process (which can be easily sampled given $x_0$ ) and the noise predicted by the network $\epsilon_\theta$ :

$L_{simple}(\theta) = \mathbb{E}_{t \sim [1, T], x_0 \sim p_{data}, \epsilon \sim \mathcal{N}(0, \mathbf{I})} \left[ || \epsilon - \epsilon_\theta(x_t, t) ||^2 \right]$

where $x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\epsilon$ is generated using the direct forward sampling formula. Training involves repeatedly sampling a data point $x_0$ , a timestep $t$ , sampling noise $\epsilon$ , constructing the corresponding $x_t$ , and performing a gradient descent step on this loss.

Once the model $\epsilon_\theta(x_t, t)$ is trained, generation (sampling) proceeds by:

Sampling initial noise $x_T \sim \mathcal{N}(0, \mathbf{I})$ .
Iterating backwards from $t = T$ $t = T$ down to $t = 1$ $t = 1$ :
- Use the network $\epsilon_\theta(x_t, t)$ to predict the noise component.
- Use this prediction to calculate the parameters (mean $\mu_\theta(x_t, t)$ and variance $\Sigma_\theta(x_t, t)$ ) of the distribution $p_\theta(x_{t-1} | x_t)$ .
- Sample $x_{t-1} \sim p_\theta(x_{t-1} | x_t)$ .
The final output $x_0$ is the generated sample.

Diagram comparing the fixed forward noising process $q$ with the learned reverse denoising process $p_\theta$ . The reverse process starts from noise $x_T$ and uses a trained model $\epsilon_\theta$ (optimized via the training objective $L_{simple}$ ) to iteratively sample less noisy states, ultimately producing a generated sample $x_0$ .

This iterative noise-and-denoise framework forms the basis of diffusion models. While conceptually straightforward, the specific choices of variance schedules, network architectures (often U-Nets), and training objectives lead to different model variants like Denoising Diffusion Probabilistic Models (DDPMs) and score-based models, which we will explore in detail in Chapter 4. Understanding this core mechanism is essential before implementing and optimizing these advanced generative techniques.