While standard Generative Adversarial Networks (GANs) learn to generate samples mimicking a data distribution, they offer little control over the specific output produced. Given a noise vector $z$ , a vanilla GAN generates an image $G(z)$ that looks like it belongs to the training dataset, but you can't easily specify which kind of image you want. Conditional GANs (cGANs) extend the GAN framework to address this limitation by incorporating auxiliary information, or conditions, into the generation process. This allows for targeted synthesis based on specific attributes.

The core idea is to provide both the generator and the discriminator with some extra information $y$ , which could represent a class label, descriptive text, or even another image. The generator's task becomes producing realistic samples that match the condition $y$ , while the discriminator must learn to distinguish real (image, condition) pairs from fake (generated image, condition) pairs.

Architecture Modification

In a standard GAN, the generator maps a random noise vector $z$ to an output image $G(z)$ . The discriminator takes an image $x$ (either real or generated) and outputs a probability $D(x)$ indicating whether it believes $x$ is real.

In a cGAN, the inputs are modified:

Generator: Takes both the noise vector $z$ and the conditional information $y$ as input. It generates an image $G(z, y)$ that should be consistent with the condition $y$ .
Discriminator: Takes both an image $x$ (real or generated) and the corresponding conditional information $y$ as input. It outputs a probability $D(x, y)$ representing the likelihood that $x$ is a real image matching the condition $y$ .

A simplified view of the Conditional GAN architecture. The condition y is provided as an additional input to both the Generator G and the Discriminator D.

The way the condition $y$ is incorporated depends on its nature and the network architecture. For discrete labels (like digits 0-9 in MNIST), $y$ can be represented as a one-hot vector and concatenated directly to $z$ for the generator. For the discriminator, it might be reshaped into a feature map and concatenated channel-wise to the image input, or embedded and combined with intermediate feature layers. For more complex conditions like text descriptions or images, embedding layers or separate encoder networks are often used to process $y$ into a suitable vector representation before combination.

Conditional Loss Function

The objective function of a cGAN adapts the standard minimax game to include the condition $y$ . The value function $V(D, G)$ becomes:

\min_G \max_D V(D, G) = \mathbb{E}_{(x, y) \sim p_{data}(x, y)}[\log D(x, y)] + \mathbb{E}_{z \sim p_z(z), y \sim p_y(y)}[\log(1 - D(G(z, y), y))]

Here:

$\mathbb{E}_{(x, y) \sim p_{data}(x, y)}[\log D(x, y)]$ represents the discriminator's objective to correctly identify real images $x$ paired with their true conditions $y$ . The expectation is taken over real data pairs $(x, y)$ drawn from the true data distribution $p_{data}$ .
$\mathbb{E}_{z \sim p_z(z), y \sim p_y(y)}[\log(1 - D(G(z, y), y))]$ represents the discriminator's objective to identify generated images $G(z, y)$ as fake, given the condition $y$ . The generator $G$ simultaneously tries to minimize this term, making its output $G(z, y)$ indistinguishable from real images for the given condition $y$ . The noise $z$ is drawn from its prior distribution $p_z$ , and the condition $y$ is typically drawn from its marginal distribution $p_y$ (or sampled alongside $x$ from the joint data distribution).

During training, mini-batches consist of pairs: (real image $x$ , condition $y$ ) and (noise vector $z$ , condition $y$ ). The generator uses $(z, y)$ to produce $G(z, y)$ . The discriminator is trained on both (real image $x$ , condition $y$ ) aiming for output close to 1, and (generated image $G(z, y)$ , condition $y$ ) aiming for output close to 0. The generator is trained based on the discriminator's output for (generated image $G(z, y)$ , condition $y$ ), aiming to make the discriminator output 1.

Applications of Conditional GANs

Conditional generation opens up numerous possibilities:

Class-Conditional Image Synthesis: Generating images of specific categories, like creating images of a particular dog breed or a specific digit. Here, $y$ is the class label.
Text-to-Image Synthesis: Generating images based on textual descriptions. The condition $y$ is derived from an embedding of the input text. This allows for creating images from sentences like "a red bird sitting on a branch".
Image-to-Image Translation: Transforming an image from one domain to another, such as converting sketches to photographs, satellite images to maps, or changing artistic styles. In this case, the condition $y$ is the input image itself (e.g., the sketch), and $G(z, y)$ produces the translated output (e.g., the photograph). Architectures like Pix2Pix are prominent examples.
Attribute Manipulation: Modifying specific attributes of an image, like changing hair color or adding glasses to a face, by conditioning on attribute vectors.

By incorporating conditional information $y$ , cGANs provide significantly more control over the generation process compared to their unconditional counterparts. This makes them a powerful tool for tasks requiring targeted synthesis or transformation of visual data. The challenge often lies in effectively integrating the condition $y$ into the network architectures and ensuring the generator properly utilizes this information to produce relevant and high-quality outputs.