Mathematical Formulation per Step

The forward diffusion process gradually adds noise to data over many timesteps, forming a Markov chain. This explanation focuses on the precise mathematical definition of a single step in this chain. It details how a state $x_{t-1}$ at timestep $t-1$ evolves into the next state $x_t$ at timestep $t$.

The transition is defined by the conditional probability distribution $q(x_t | x_{t-1})$. In standard diffusion models like Denoising Diffusion Probabilistic Models (DDPM), this transition is modeled as adding a small amount of Gaussian noise. The amount of noise added at each step is controlled by a predetermined variance schedule, denoted by $\beta_t$, where $t$ ranges from 1 to $T$ (the total number of diffusion steps).

Specifically, the distribution $q(x_t | x_{t-1})$ is defined as a Gaussian distribution whose mean depends on the previous state $x_{t-1}$ and whose variance is given by $\beta_t$:

$$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t I)$$

Let's break down this equation:

• $x_t$: The data (e.g., an image tensor) at timestep $t$.
• $x_{t-1}$: The data at the previous timestep $t-1$.
• $\beta_t$: The variance of the Gaussian noise added at step $t$. This is a small positive number determined by the variance schedule. Typical values might start very small (e.g., $10^{-4}$) and increase slightly over the steps.
• $I$: The identity matrix. This is relevant because $x_t$ is usually multi-dimensional (e.g., pixels in an image). The noise is added independently to each dimension with the same variance $\beta_t$.
• $\mathcal{N}(x; \mu, \Sigma)$: This denotes a Gaussian (Normal) distribution over variable $x$ with mean $\mu$ and covariance matrix $\Sigma$. In our case, the mean is $\sqrt{1 - \beta_t} x_{t-1}$ and the covariance is $\beta_t I$.

This formula tells us that $x_t$ is centered around a slightly scaled-down version of $x_{t-1}$ (scaled by $\sqrt{1 - \beta_t}$), with added noise controlled by $\beta_t$. Because $\beta_t$ is small, $\sqrt{1 - \beta_t}$ is slightly less than 1, so the signal from $x_{t-1}$ is mostly preserved, but noise is introduced.

For convenience, it's common to define $\alpha_t = 1 - \beta_t$. Since $\beta_t$ is small and positive, $\alpha_t$ is slightly less than 1. Using this notation, the equation becomes:

$$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{\alpha_t} x_{t-1}, (1 - \alpha_t) I)$$

This form highlights that the new state $x_t$ is a combination of the scaled previous state $\sqrt{\alpha_t} x_{t-1}$ and newly added noise with variance $1 - \alpha_t = \beta_t$.

We can express the process of sampling $x_t$ from $x_{t-1}$ using the reparameterization trick. If $\epsilon_{t-1}$ is a random variable drawn from a standard Gaussian distribution $\mathcal{N}(0, I)$, then we can write $x_t$ as:

$$x_t = \sqrt{\alpha_t} x_{t-1} + \sqrt{1 - \alpha_t} \epsilon_{t-1}$$

Here, $\epsilon_{t-1} \sim \mathcal{N}(0, I)$. This formulation is particularly useful for implementation, as it clearly separates the deterministic part (scaling $x_{t-1}$) and the stochastic part (adding scaled standard Gaussian noise).

The sequence of values $\beta_1, \beta_2, ..., \beta_T$ (or equivalently $\alpha_1, \alpha_2, ..., \alpha_T$) constitutes the noise schedule. The choice of this schedule is an important design decision that affects the diffusion process and model performance. We will examine different scheduling strategies in the next section.

For now, the important takeaway is this single-step transition formula. It's the fundamental building block of the entire forward diffusion process, defining precisely how noise is incrementally added at each step of the Markov chain. Understanding this equation is necessary for grasping both the forward process properties and how the reverse (denoising) process is formulated later.

Let's visualize this for a single data point (1-dimensional) transitioning from $x_{t-1}$ to $x_t$.

Diagram illustrating a single step $x_{t-1} \rightarrow x_t$. The point $x_{t-1}$ is scaled down to $\sqrt{\alpha_t} x_{t-1}$, and then Gaussian noise with variance $\beta_t = 1 - \alpha_t$ is added, resulting in the new state $x_t$.

This step-by-step addition of controlled noise ensures that if we repeat this process for $T$ steps, the resulting $x_T$ will closely resemble pure noise, effectively destroying the original data structure. The next section will discuss how the $\beta_t$ values are chosen in the noise schedule, and later sections will show how we can derive a formula to jump directly from $x_0$ to any $x_t$.