Gaussian Noise Schedule

The forward diffusion process is a Markov chain that gradually injects noise into data $x_0$ over $T$ discrete timesteps. A significant aspect of this process is controlling how much noise is added at each step. This isn't done with random amounts; instead, a carefully defined plan known as the noise schedule is followed.

The noise schedule is a sequence of variance values, denoted as $\beta_1, \beta_2, ..., \beta_T$. Each $\beta_t$ determines the variance of the Gaussian noise added when transitioning from state $x_{t-1}$ to $x_t$. These values are hyperparameters, meaning they are chosen before training the model. They are not learned during the training process itself.

Think of the noise schedule as setting the "intensity" of the noising process at each step. The values of $\beta_t$ are typically chosen such that:

1. They are small, ensuring that the change between consecutive steps $x_{t-1}$ and $x_t$ is limited.
2. They increase gradually over the timesteps, so more noise is added in later steps compared to earlier ones.
3. The cumulative effect over $T$ steps is sufficient to transform the original data distribution into something very close to a standard Gaussian distribution (isotropic noise).

Defining the Schedule Values

The original Denoising Diffusion Probabilistic Models (DDPM) paper proposed a linear schedule, where $\beta_t$ increases linearly from a small starting value $\beta_{start}$ (e.g., $10^{-4}$) to a larger ending value $\beta_{end}$ (e.g., $0.02$) over $T$ steps (often $T=1000$).

The formula for a linear schedule is:

$$\beta_t = \beta_{start} + (t-1) \frac{\beta_{end} - \beta_{start}}{T-1} \quad \text{for } t = 1, ..., T$$

While simple and effective, other schedules have been developed. A popular alternative is the cosine schedule, introduced in the "Improved Denoising Diffusion Probabilistic Models" paper. This schedule changes more slowly near the beginning and end of the process, potentially leading to better performance and preventing the signal from being destroyed too quickly early on.

The cosine schedule is defined using related quantities $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{i=1}^t \alpha_i$. It sets $\bar{\alpha}_t$ based on a cosine function and then derives the corresponding $\beta_t$:

$$\bar{\alpha}_t = \frac{f(t)}{f(0)} \quad \text{where} \quad f(t) = \cos\left( \left( \frac{t}{T} + s \right) / (1+s) \cdot \frac{\pi}{2} \right)^2$$

Here, $s$ is a small offset (e.g., $0.008$) to prevent $\beta_t$ from being too small near $t=0$. Once $\bar{\alpha}_t$ is calculated for all $t$, the individual $\beta_t$ values can be recovered:

$$\beta_t = 1 - \frac{\bar{\alpha}_t}{\bar{\alpha}_{t-1}}$$

The choice of schedule impacts how quickly information from the original data $x_0$ is obscured. A schedule that adds noise too aggressively early on might make the reverse process harder to learn. Conversely, adding too little noise overall might not sufficiently transform the data into a simple prior distribution by step $T$.

Let's visualize how these two common schedules compare for $T=1000$ steps, with $\beta_{start}=0.0001$ and $\beta_{end}=0.02$ for the linear schedule, and $s=0.008$ for the cosine schedule derived to approximately match the noise level at $T$.

Variance ($\beta_t$) added at each timestep for linear and cosine schedules over $T=1000$ steps. The cosine schedule adds noise more slowly initially and accelerates towards the end compared to the linear schedule.

Role in the Markov Step

Recall from the previous section that the forward process step is defined by the conditional probability $q(x_t | x_{t-1})$. This transition is defined as adding Gaussian noise with a specific mean and variance:

$$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t \mathbf{I})$$

Here, the noise schedule value $\beta_t$ directly sets the variance of the Gaussian noise added at timestep $t$. The mean is scaled by $\sqrt{1 - \beta_t}$ to ensure the overall variance of the data doesn't explode. Since $\beta_t$ is small, $\sqrt{1 - \beta_t}$ is slightly less than 1, gradually shrinking the contribution of the previous state $x_{t-1}$.

In summary, the Gaussian noise schedule $\{\beta_t\}_{t=1}^T$ is a sequence of pre-defined hyperparameters controlling the magnitude of noise added at each step of the forward diffusion process. Its design (e.g., linear, cosine) and the range of values are important choices that influence the trajectory from data to noise and affect the subsequent learning of the reverse process. We will see later how these $\beta_t$ values, and related quantities derived from them, appear in both the training objective and the sampling process.