Classifier-Free Guidance (CFG)

Steering the diffusion process for conditional generation often requires training and maintaining a separate classifier model alongside the diffusion model. This approach adds complexity and potential points of failure. Furthermore, the gradients from such a classifier might not always align perfectly with the diffusion model's internal representation, sometimes leading to artifacts or suboptimal results.

Classifier-Free Guidance (CFG) offers an elegant and effective alternative that achieves conditional generation without needing an external classifier. The main idea is to train the diffusion model itself to handle both conditional and unconditional generation scenarios.

How CFG Training Works

To enable CFG, the noise prediction network, typically a U-Net denoted as $\epsilon_\theta(x_t, t, y)$, is trained on both conditional and unconditional inputs. During training, the conditioning information $y$ (like a class label or text embedding) is randomly dropped or replaced with a special "null" token $\emptyset$ some percentage of the time (e.g., 10-20% of training examples).

So, the model learns to predict:

1. The noise given the condition: $\epsilon_\theta(x_t, t, y)$
2. The noise without any condition: $\epsilon_\theta(x_t, t, \emptyset)$

This joint training forces the model to understand the difference between generating data specific to condition $y$ and generating data typical of the overall distribution. The same network parameters $\theta$ are used for both tasks. This means the U-Net architecture must be adapted to accept the conditioning information $y$ as an additional input, alongside the noisy image $x_t$ and the timestep $t$.

CFG Sampling: Guiding the Denoising Step

During the sampling (reverse diffusion) process, we leverage the model's ability to predict both conditional and unconditional noise. At each denoising step $t$, we perform two forward passes through the U-Net using the current noisy state $x_t$ and timestep $t$:

• One pass provides the conditional noise prediction: $\epsilon_\theta(x_t, t, y)$, using the desired condition $y$.
• Another pass provides the unconditional noise prediction: $\epsilon_\theta(x_t, t, \emptyset)$, using the null token $\emptyset$ in place of the condition.

The intuition is that the difference between these two predictions, $\epsilon_\theta(x_t, t, y) - \epsilon_\theta(x_t, t, \emptyset)$, represents the direction in the noise space that moves the generation towards satisfying the condition $y$. CFG combines these predictions to create an adjusted noise estimate $\hat{\epsilon}_\theta$ that extrapolates further in the conditional direction.

The formula for the CFG-adjusted noise prediction is:

$$\hat{\epsilon}_\theta(x_t, t, y, w) = \epsilon_\theta(x_t, t, \emptyset) + w (\epsilon_\theta(x_t, t, y) - \epsilon_\theta(x_t, t, \emptyset))$$

Here, $w$ is the guidance scale.

• If $w = 0$, the formula simplifies to $\hat{\epsilon}_\theta = \epsilon_\theta(x_t, t, \emptyset)$. This corresponds to purely unconditional generation, ignoring $y$.
• If $w = 1$, we get $\hat{\epsilon}_\theta = \epsilon_\theta(x_t, t, y)$. This is equivalent to standard conditional generation using only the conditional prediction without any explicit guidance boost.
• If $w > 1$, the process extrapolates. It starts with the unconditional prediction $\epsilon_\theta(x_t, t, \emptyset)$ and moves further in the direction indicated by the condition $y$ (represented by the difference vector), scaled by $w$. This effectively amplifies the influence of the condition $y$ on the generation process.

This adjusted noise estimate $\hat{\epsilon}_\theta(x_t, t, y, w)$ is then used within the denoising update rule (e.g., in DDPM or DDIM sampling) to calculate the estimate for the less noisy state $x_{t-1}$.

The Guidance Scale (w)

The guidance scale $w$ is a hyperparameter that allows you to control the strength of the conditioning. It acts as a knob balancing adherence to the condition against sample diversity and quality.

• Low $w$ (e.g., 1-3): Weak guidance. Samples generated tend to be more diverse and creative but might only loosely follow the provided condition $y$.
• Moderate $w$ (e.g., 5-10): Often provides a good balance. Samples usually adhere well to the condition $y$ while maintaining reasonable visual quality and some degree of diversity. This range is frequently used in practice for text-to-image models.
• High $w$ (e.g., 15+): Strong guidance. Samples very closely follow the condition $y$. However, this can sometimes lead to reduced diversity (samples look similar), potential oversaturation of colors or features, or other artifacts as the extrapolation pushes the generation process into areas less explored during training.

Choosing the right value for $w$ often requires some experimentation based on the specific diffusion model, the dataset it was trained on, and the desired output characteristics.

A diagram illustrating how Classifier-Free Guidance combines the unconditional noise prediction $\epsilon_\theta(x_t, \emptyset)$ and the conditional prediction $\epsilon_\theta(x_t, y)$ to produce an adjusted prediction $\hat{\epsilon}_\theta(x_t, y, w)$ by extrapolating along the difference vector, controlled by the guidance scale $w$. This example shows extrapolation for $w=2$.

Advantages of CFG

Classifier-Free Guidance has become a standard technique for conditional diffusion models due to several significant advantages:

1. No Extra Classifier: It eliminates the need to train, manage, and ensure compatibility of a separate classification model. This simplifies the overall pipeline.
2. Simplicity: The implementation primarily involves modifying the training data (by randomly dropping conditions) and the sampling loop (by computing two forward passes through the U-Net per step and combining the results using the CFG formula).
3. Effectiveness: CFG often yields high-quality conditional samples that strongly adhere to the provided guidance. It generally performs as well as or better than classifier guidance across various tasks, particularly in text-to-image generation.
4. Flexibility: The guidance scale $w$ provides a simple and intuitive way to control the trade-off between condition adherence and sample diversity at inference time, without needing to retrain the model.

By integrating the conditioning mechanism directly into the diffusion model's training objective, CFG provides a powerful and widely adopted method for controlling the generation process. We will explore how to implement this technique in practice later in this chapter.