Implementing Classifier-Free Guidance

Classifier-Free Guidance (CFG) offers a powerful way to steer the generation process without relying on a separate, pre-trained classifier model. This technique avoids potential issues tied to classifier accuracy or domain mismatch, becoming a standard method in modern diffusion models.

The core idea is to train a single diffusion model $\epsilon_\theta$ that can operate both conditionally and unconditionally. This is achieved through a modification during the training phase.

Training with Conditional Dropout

During training, for each data sample $x_0$ and its corresponding conditioning information $y$ (like a class label or text embedding), we do the following:

1. Sample a timestep $t \sim \mathcal{U}(1, T)$.
2. Sample noise $\epsilon \sim \mathcal{N}(0, \mathbf{I})$.
3. Compute the noisy sample $x_t$ using the forward process equation: $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon$.
4. With a certain probability $p_{uncond}$ (e.g., 10-20%), replace the actual conditioning information $y$ with a special null or unconditional token, denoted as $\emptyset$. This replacement acts like a form of "conditional dropout".
5. Feed the noisy sample $x_t$, the timestep $t$, and the (potentially replaced) conditioning information into the U-Net model: $\epsilon_{pred} = \epsilon_\theta(x_t, t, y_{eff})$, where $y_{eff}$ is either $y$ or $\emptyset$.
6. Calculate the loss, typically the Mean Squared Error (MSE), between the predicted noise $\epsilon_{pred}$ and the true noise $\epsilon$: $L = ||\epsilon - \epsilon_{pred}||^2$.
7. Update the model parameters $\theta$ using gradient descent on this loss.

By randomly omitting the conditioning information during training, the model learns two things simultaneously:

• How to predict the noise given the condition: $\epsilon_\theta(x_t, t, y)$.
• How to predict the noise unconditionally: $\epsilon_\theta(x_t, t, \emptyset)$.

The null token $\emptyset$ needs a specific representation. For class labels, it might be a dedicated "unconditional" class index. For text embeddings (like CLIP embeddings), it's often a fixed, learnable embedding vector trained alongside the model to represent the absence of text conditioning.

Sampling with Guidance

During the generation (sampling) process, we leverage the model's ability to make both conditional and unconditional predictions at each step. For a given timestep $t$ and the current noisy sample $x_t$, we compute two noise predictions:

1. Unconditional Prediction: $\epsilon_{uncond} = \epsilon_\theta(x_t, t, \emptyset)$
2. Conditional Prediction: $\epsilon_{cond} = \epsilon_\theta(x_t, t, y)$ (where $y$ is the desired condition for the output)

Instead of using just $\epsilon_{cond}$ to perform the denoising step, we combine these two predictions using a guidance scale parameter $w$ (often called guidance strength or scale, sometimes denoted $s$ or $\gamma$). The combined noise prediction $\tilde{\epsilon}_t$ is calculated as:

$$\tilde{\epsilon}_t = \epsilon_{uncond} + w \cdot (\epsilon_{cond} - \epsilon_{uncond})$$

This formula has a clear interpretation:

• Start with the unconditional noise prediction $\epsilon_{uncond}$.
• Calculate the "guidance direction": the difference between the conditional and unconditional predictions $(\epsilon_{cond} - \epsilon_{uncond})$. This vector points from the unconditional generation path towards the conditional one.
• Scale this direction by the guidance scale $w$ and add it to the unconditional prediction.

An equivalent way to write this is:

$$\tilde{\epsilon}_t = (1-w)\epsilon_{uncond} + w \cdot \epsilon_{cond}$$

From this form, we can see:

• If $w = 0$, we get $\tilde{\epsilon}_t = \epsilon_{uncond}$, resulting in purely unconditional generation.
• If $w = 1$, we get $\tilde{\epsilon}_t = \epsilon_{cond}$, which corresponds to standard conditional generation without extra guidance amplification.
• If $w > 1$, we extrapolate along the guidance direction, pushing the generation more strongly towards the condition $y$.

This combined noise estimate $\tilde{\epsilon}_t$ is then used in the standard denoising step (e.g., the DDPM or DDIM update rule) to compute the less noisy sample $x_{t-1}$. The process repeats from $t=T$ down to $t=1$.

The diagram below illustrates the computation at a single sampling step:

Flow diagram for calculating the guided noise prediction $\tilde{\epsilon}_t$ using Classifier-Free Guidance at a single denoising step $t$.

Practical Notes

• Guidance Scale ($w$): This is a critical hyperparameter. Typical values range from $1.5$ to $15$. Higher values enforce the condition more strongly, which can improve alignment (e.g., making an image look more like the text prompt) but may lead to saturation, artifacts, or reduced diversity in the generated samples. Lower values yield more diverse but potentially less condition-aligned results. You often need to experiment to find a good balance for your specific task and model.
• Conditioning Input: How $y$ and $\emptyset$ are integrated into the U-Net architecture is important. Common methods include adding their embeddings to the timestep embeddings or using cross-attention layers within the U-Net blocks, allowing the model to attend to relevant parts of the conditioning information. We'll discuss architecture modifications in more detail later.
• Computational Cost: CFG requires running the model forward twice per sampling step (once conditional, once unconditional). This roughly doubles the computational cost of sampling compared to standard conditional or unconditional generation. This trade-off is often acceptable given the significant improvement in control and quality.

Implementing CFG involves modifying both the training loop (to handle conditional dropout) and the sampling loop (to perform the two forward passes and combine the results). The next sections will explore specific architectural changes often used for conditioning and provide practical examples.