In the previous chapter, we established the forward process, a fixed procedure for gradually adding noise to data. Our generative goal, however, is to reverse this: starting from noise $x_T \sim \mathcal{N}(0, \mathbf{I})$ , we want to sample $x_{T-1}$ , then $x_{T-2}$ , and so on, until we arrive at a sample $x_0$ that looks like it came from our original data distribution. This requires calculating the reverse transition probabilities, $p(x_{t-1}|x_t)$ .

If we could compute $p(x_{t-1}|x_t)$ exactly, we could sample from it iteratively to generate data. However, calculating this distribution directly poses a significant challenge. Why? Because the probability of transitioning back to a less noisy state $x_{t-1}$ depends not just on the current noisy state $x_t$ , but implicitly on the entire distribution of possible starting data points $q(x_0)$ . Mathematically, evaluating $p(x_{t-1}|x_t)$ involves marginalizing over all possible initial data points $x_0$ :

p(x_{t-1}|x_t) = \int p(x_{t-1}|x_t, x_0) q(x_0|x_t) dx_0

The term $q(x_0|x_t)$ represents the probability of a specific starting data point $x_0$ given the noisy version $x_t$ . Calculating this requires knowledge of the (unknown) true data distribution $q(x_0)$ and involves complex integration, making the true reverse probability $p(x_{t-1}|x_t)$ intractable to compute for complex datasets.

The Tractable Conditional Reverse Distribution

Interestingly, the situation changes if we know the starting point $x_0$ that led to $x_t$ . The reverse conditional probability $p(x_{t-1}|x_t, x_0)$ is tractable. Using Bayes' theorem and the properties of the Gaussian noise added during the forward process $q(x_t|x_{t-1})$ (defined by the noise schedule $\beta_t$ ), we can show that this distribution is also a Gaussian:

p(x_{t-1}|x_t, x_0) = \mathcal{N}(x_{t-1}; \tilde{\mu}_t(x_t, x_0), \tilde{\beta}_t \mathbf{I})

Here, $\mathcal{N}(x; \mu, \sigma^2 \mathbf{I})$ denotes a Gaussian distribution with mean $\mu$ and diagonal covariance $\sigma^2 \mathbf{I}$ . The variance $\tilde{\beta}_t$ and the mean $\tilde{\mu}_t(x_t, x_0)$ depend only on the known forward process noise schedule parameters ( $\beta_t$ or equivalently $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ ) and the specific values of $x_t$ and $x_0$ . Specifically:

\tilde{\beta}_t = \frac{1 - \bar{\alpha}_{t-1}}{1 - \bar{\alpha}_t} \beta_t

\tilde{\mu}_t(x_t, x_0) = \frac{\sqrt{\bar{\alpha}_{t-1}} \beta_t}{1 - \bar{\alpha}_t} x_0 + \frac{\sqrt{\alpha_t} (1 - \bar{\alpha}_{t-1})}{1 - \bar{\alpha}_t} x_t

These equations tell us that if we knew the original image $x_0$ corresponding to a noisy image $x_t$ , we could precisely calculate the distribution of the slightly less noisy image $x_{t-1}$ it came from.

The Need for Approximation

But here's the catch: during the actual generative process (sampling), we don't have $x_0$ . Our goal is precisely to generate $x_0$ starting from pure noise $x_T$ . Knowing $x_0$ would defeat the purpose.

Since the true reverse transition $p(x_{t-1}|x_t)$ is intractable, and the conditional reverse transition $p(x_{t-1}|x_t, x_0)$ requires the unknown $x_0$ , we need an alternative. The solution is to approximate the true reverse transition using a learned model.

We introduce a parameterized distribution, typically a neural network, denoted as $p_\theta(x_{t-1}|x_t)$ , to approximate the intractable true posterior $p(x_{t-1}|x_t)$ . The parameters $\theta$ of this network will be learned from data.

Diagram illustrating the relationship between the intractable true reverse transition, the tractable conditional reverse transition (requiring $x_0$ ), and the learned approximation $p_\theta(x_{t-1}|x_t)$ using a neural network.

This neural network takes the current noisy state $x_t$ and the current timestep $t$ as input and outputs the parameters of the approximate reverse distribution $p_\theta(x_{t-1}|x_t)$ . Since the tractable conditional reverse distribution $p(x_{t-1}|x_t, x_0)$ is Gaussian, it's convenient and effective to also model our approximation $p_\theta(x_{t-1}|x_t)$ as a Gaussian:

p_\theta(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))

Here, $\mu_\theta(x_t, t)$ is the mean and $\Sigma_\theta(x_t, t)$ is the covariance matrix, both predicted by the neural network $\theta$ . The network's task is therefore to learn functions $\mu_\theta$ and $\Sigma_\theta$ that accurately predict the parameters of the distribution for $x_{t-1}$ given $x_t$ .

In many successful diffusion models, like the original Denoising Diffusion Probabilistic Models (DDPM), the covariance matrix $\Sigma_\theta(x_t, t)$ is not learned directly. Instead, it's often fixed to a value related to the forward process variance, such as $\Sigma_\theta(x_t, t) = \tilde{\beta}_t \mathbf{I}$ or $\Sigma_\theta(x_t, t) = \beta_t \mathbf{I}$ . This simplifies the learning problem significantly: the neural network only needs to learn the mean $\mu_\theta(x_t, t)$ of the reverse transition.

The next step is to understand how we parameterize this neural network, specifically how it predicts the mean $\mu_\theta(x_t, t)$ , and how we train it to effectively reverse the diffusion process. As we will see, a common and effective strategy is to train the network to predict the noise that was added at timestep $t$ .