Properties of the Forward Process

The forward diffusion process has important characteristics. Understanding these properties is primary for grasping how diffusion models function and why their design choices are made.

Gaussian Nature of the Process

Recall from the previous sections that each step of the forward process adds Gaussian noise. Specifically, the transition distribution is defined as:

$$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t I)$$

where $\beta_t$ is the variance schedule at timestep $t$, and $I$ is the identity matrix. Because we start with data $x_0$ and repeatedly add Gaussian noise, the marginal distribution of any noisy sample $x_t$ conditioned on the starting point $x_0$ is also Gaussian. As derived earlier, this distribution has a convenient closed form:

$$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1 - \bar{\alpha}_t) I)$$

Here, $\alpha_t = 1 - \beta_t$, and $\bar{\alpha}_t = \prod_{i=1}^t \alpha_i$. This property is very useful because it means we can directly sample $x_t$ from $x_0$ for any timestep $t$ without iterating through all the intermediate steps $x_1, x_2, ..., x_{t-1}$. This significantly speeds up the training process later on.

Convergence to a Standard Gaussian Distribution

The primary goal of the forward process is to gradually transform the complex data distribution $q(x_0)$ into a simple, known distribution, typically an isotropic Gaussian distribution $\mathcal{N}(0, I)$. Does our defined process achieve this?

Let's look again at the distribution $q(x_t | x_0)$:

$$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1 - \bar{\alpha}_t) I)$$

The noise schedule $\{\beta_t\}_{t=1}^T$ is typically designed such that $\beta_t$ values are small but positive. This ensures that $\alpha_t = 1 - \beta_t$ is slightly less than 1. Consequently, the cumulative product $\bar{\alpha}_t = \prod_{i=1}^t \alpha_i$ is a value that starts at $\bar{\alpha}_0 = 1$ (by convention) and steadily decreases as $t$ increases.

For a sufficiently large number of steps $T$ (e.g., $T=1000$ or more) and a suitable schedule $\{\beta_t\}$, the value of $\bar{\alpha}_T$ becomes very close to zero.

• As $\bar{\alpha}_T \rightarrow 0$, the mean term $\sqrt{\bar{\alpha}_T} x_0$ approaches $0$.
• As $\bar{\alpha}_T \rightarrow 0$, the variance term $(1 - \bar{\alpha}_T) I$ approaches $(1 - 0) I = I$.

Therefore, for large $T$, the distribution $q(x_T | x_0)$ becomes:

$$q(x_T | x_0) \approx \mathcal{N}(x_T; 0, I)$$

This means that after $T$ steps, the resulting sample $x_T$ is essentially pure Gaussian noise, and almost all information about the original data point $x_0$ has been destroyed. The forward process successfully converts any input data point into a sample from a standard Gaussian distribution, regardless of the starting point $x_0$.

The value of $\bar{\alpha}_t$ decreases from 1 towards 0 as the timestep $t$ increases, indicating the diminishing influence of the original data $x_0$ and the increasing dominance of noise. Values shown are illustrative for a typical schedule over $T=1000$ steps.

Tractability and Fixed Nature

A significant property of the forward process is its tractability. As mentioned, we can calculate the distribution $q(x_t | x_0)$ directly using the closed-form expression. This allows us to efficiently sample $x_t$ for any $t$ given $x_0$, which is essential for training the neural network that will learn the reverse process. We don't need to simulate the step-by-step noising during training.

Furthermore, the entire forward process is fixed. It does not involve any learnable parameters. The noise schedule $\{\beta_t\}_{t=1}^T$ is chosen beforehand (e.g., a linear schedule, cosine schedule) and remains constant throughout training and inference. All the learning happens in the reverse process, which must learn to undo this fixed noising procedure.

Markov Property

As defined, the forward process is a Markov chain. This means that the distribution of the state $x_t$ only depends on the immediately preceding state $x_{t-1}$, not on any earlier states $x_0, ..., x_{t-2}$.

$$q(x_t | x_{t-1}, x_{t-2}, ..., x_0) = q(x_t | x_{t-1})$$

While we derived a useful expression for $q(x_t | x_0)$, the underlying step-by-step process retains this Markov property. This structure simplifies the mathematical analysis and is mirrored (though approximated) in the reverse process.

In summary, the forward process is a fixed, tractable mechanism that gradually and controllably converts data into noise following Gaussian statistics. Its endpoint is designed to be a simple, known distribution ($\mathcal{N}(0, I)$), and its intermediate steps $q(x_t | x_0)$ are easily calculated. These properties form the foundation upon which the learnable reverse (denoising) process is built.