Motivation for Conditional Generation

Diffusion models excel at learning the underlying distribution of a dataset $p(x)$ and generating high-fidelity samples by reversing a gradual noising process. Starting from pure Gaussian noise $x_T$, the reverse process iteratively denoises it to produce a sample $x_0$ that looks like it came from the original data.

However, this standard generation process is unconditional. While it produces realistic outputs, it doesn't offer explicit control over what specific kind of output is generated. If you train a diffusion model on a diverse dataset of animals, running the standard sampling procedure might yield an image of a dog, a cat, a bird, or any other animal present in the training data. You effectively get a random sample from the learned distribution $p(x)$, but you cannot directly ask the model to generate, say, only images of cats.

This lack of direct control limits the practical applicability of unconditional models in many scenarios. Often, we need to guide the generation process based on specific requirements or inputs. Consider these common use cases:

• Class-Conditional Generation: We might want to generate an image belonging to a specific category, like generating only pictures of "bridges" from a dataset, or generating only "sedan" type cars.
• Text-to-Image Synthesis: A highly popular application involves generating images based on descriptive text prompts, such as creating an image for "an astronaut riding a horse on the moon."
• Image Editing and Translation: Tasks like image inpainting (filling missing regions), style transfer (applying the style of one image to another), or super-resolution often require conditioning the generation on an existing input image.
• Data Augmentation: Generating synthetic data that meets certain specific criteria for training downstream models.

In all these examples, the goal is not just to sample from the overall data distribution $p(x)$, but rather to sample from a conditional distribution $p(x|y)$, where $y$ represents the conditioning information. This conditioning variable $y$ could be:

• A class label (e.g., an integer representing "cat" or "dog").
• A text embedding derived from a prompt.
• Another image (for tasks like image translation or editing).
• Other forms of structured data relevant to the generation task.

Therefore, we need mechanisms to incorporate this conditioning information $y$ into the diffusion model's generation process. We need ways to steer the iterative denoising steps so that the final output $x_0$ not only looks realistic (belongs to the data manifold) but also aligns with the provided condition $y$.

This chapter focuses on precisely these mechanisms. We will investigate techniques that allow us to exert control over the diffusion model's output, transforming it from a generator of random samples into a controllable synthesis engine. We'll start by looking at how external models can guide the process and then move to more integrated approaches like classifier-free guidance, which has become a standard technique for conditional diffusion models.