Implementing and Tuning CFG Scale

Classifier-Free Guidance (CFG) is a method for controlling generation in diffusion models. This discussion focuses on its practical implementation and the process of tuning the guidance scale, often denoted as $s$ or $w$. The guidance scale dictates how strongly the generation process adheres to the provided conditioning signal (like a text prompt or class label).

Implementation Mechanism

CFG works by cleverly modifying the model's predicted noise (or predicted $x_0$, depending on the parameterization) during the sampling process. At each timestep $t$, the model makes two predictions:

1. Conditional Prediction: $\epsilon_\theta(x_t, t, c)$, where $c$ is the conditioning information (e.g., text embedding, class ID).
2. Unconditional Prediction: $\epsilon_\theta(x_t, t, \emptyset)$, where $\emptyset$ represents null or absent conditioning.

During training, this unconditional prediction is often achieved by randomly dropping the conditioning information for a fraction of the training examples (e.g., replacing text embeddings with a learned null token or class labels with a special 'unconditional' ID). This trains the model to predict noise both with and without guidance within the same architecture.

At inference time, both predictions are computed. The final noise prediction used for the denoising step is a linear combination, extrapolated away from the unconditional prediction in the direction of the conditional one:

$$\hat{\epsilon}_\theta(x_t, t, c) = \epsilon_\theta(x_t, t, \emptyset) + s \cdot (\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \emptyset))$$

Here, $s$ is the guidance scale. Notice that if $s=0$, we recover the unconditional prediction, $\hat{\epsilon}_\theta = \epsilon_\theta(x_t, t, \emptyset)$. If $s=1$, we recover the standard conditional prediction, $\hat{\epsilon}_\theta = \epsilon_\theta(x_t, t, c)$. Values of $s > 1$ push the prediction further in the direction indicated by the condition, effectively amplifying the guidance.

This combined prediction $\hat{\epsilon}_\theta$ is then used within the chosen sampler (e.g., DDIM, DPM-Solver) to estimate $x_{t-1}$ from $x_t$.

The Guidance Scale Parameter ($s$)

The guidance scale $s$ is a hyperparameter that controls the trade-off between sample quality (adherence to the condition) and diversity.

• Low $s$ (e.g., 0-3): Generation relies less on the condition.
  • $s=0$: Purely unconditional generation, ignoring $c$.
  • $s=1$: Standard conditional generation (as learned).
  • Slightly > 1: Samples are diverse, potentially more creative or abstract, but may not strictly follow the prompt or condition.
• Moderate $s$ (e.g., 4-10): A common range that often yields a good balance. Samples generally adhere well to the condition while retaining reasonable diversity. Many standard text-to-image models default to values around 7-8.
• High $s$ (e.g., 11-20+): Strong adherence to the condition. Samples can be very accurate representations of the prompt but may suffer from reduced diversity, potential saturation issues (e.g., overly bright or contrasty images), or visual artifacts as the model is pushed into less explored regions of its learned distribution.

The impact of $s$ is highly dependent on the specific model, dataset, and task. There's no single "best" value; it requires tuning.

Tuning Strategies for the Guidance Scale

Finding an effective guidance scale typically involves experimentation:

1. Visual Inspection: Generate samples using the same initial noise and conditioning but vary the scale $s$. Observe how the adherence to the condition changes and whether artifacts appear at higher scales. This is often the primary method for tasks like image generation.
2. Metric-Based Evaluation: If quantitative metrics are available (e.g., CLIP score for text-to-image alignment, classification accuracy for class-conditional generation), generate batches of samples at different scales and plot the metric against $s$. This can reveal trends but should be combined with visual inspection, as metrics don't always capture perceptual quality perfectly.
3. Trade-off Analysis: Consider plotting adherence metrics against diversity metrics (if available) for different scales. This helps visualize the compromise being made as $s$ increases.

Relationship between guidance scale ($s$) and metrics for prompt adherence, sample diversity, and the likelihood of artifacts. Increasing $s$ generally improves adherence but reduces diversity and may introduce artifacts.

Code Example Snippet

Below is a simplified illustration of how the CFG logic modifies a sampling loop. Assume model can predict noise $\epsilon_\theta$, sampler handles the denoising step, latents is $x_t$, t is the timestep, cond_embedding is the conditioning, and uncond_embedding is the null conditioning.

import torch

# Assume model, sampler, latents, t, cond_embedding, uncond_embedding are defined
# Assume guidance_scale (s) is set, e.g., s = 7.5

# Concatenate inputs for batched inference
latent_model_input = torch.cat([latents] * 2)
time_input = torch.cat([t] * 2)
context_input = torch.cat([uncond_embedding, cond_embedding])

# Predict noise for both unconditional and conditional inputs
noise_pred_uncond, noise_pred_cond = model(latent_model_input, time_input, context=context_input).chunk(2)

# Combine predictions using the CFG formula
noise_pred_cfg = noise_pred_uncond + guidance_scale * (noise_pred_cond - noise_pred_uncond)

# Use the combined prediction in the sampler's step function
latents = sampler.step(noise_pred_cfg, t, latents)

# ... rest of the sampling loop ...

Practical Notes

• Computational Cost: The primary drawback of CFG is the need for two forward passes through the model at each sampling step (one conditional, one unconditional). This roughly doubles the inference time compared to using only the conditional model or a model trained without CFG. Research into optimizing this is ongoing.
• Interaction with Samplers: The optimal s value might slightly change depending on the sampler used (e.g., DDIM vs. DPM-Solver++) and the number of sampling steps. Faster samplers using fewer steps might sometimes benefit from slightly higher guidance scales to maintain prompt fidelity.
• Training Stability: While CFG is applied during inference, the model must be trained robustly with the conditioning dropout mechanism. If the model doesn't learn the unconditional prediction well, CFG might perform poorly. Techniques discussed later, like EMA and gradient clipping, help ensure the base model is stable.

Mastering the implementation and tuning of the CFG scale is a fundamental skill for controlling modern diffusion models. It provides a powerful knob for balancing fidelity to the desired output against the inherent creativity and diversity of the generative process. Experimentation, guided by both visual feedback and relevant metrics, is essential for finding the sweet spot for your specific application.