Architectures for cGANs

To effectively control the output of a Generative Adversarial Network, we need mechanisms to incorporate conditional information, denoted as $y$, into the generation process. This condition $y$ could represent various things, such as a class label (e.g., "dog," "cat"), specific attributes (e.g., hair color, pose), a textual description, or even another image. The core idea of Conditional GANs (cGANs) is to modify both the generator $G$ and the discriminator $D$ to operate based on $y$, resulting in $G(z, y)$ and $D(x, y)$. Here are common architectural approaches to achieve this conditioning.

Conditioning the Generator

The generator's task is to produce a sample $x$ that belongs to the target data distribution and corresponds to the provided condition $y$. The latent vector $z$ still provides the source of variation, while $y$ directs the synthesis towards a specific mode or characteristic.

1. Concatenation-Based Conditioning

The most straightforward method is to directly feed the conditional information $y$ into the generator alongside the latent vector $z$.

• Input Concatenation: If $y$ is represented as a vector (e.g., a one-hot encoded class label or an embedding from text/image), it can be concatenated with the latent vector $z$. This combined vector $[z, y]$ then serves as the input to the generator's first layer.

Concatenating the latent vector $z$ and the (potentially embedded) condition vector $y$ as input to the generator network.

• Intermediate Layer Concatenation: Instead of only at the input, the condition $y$ (often after being processed through its own embedding layer or small neural network) can be reshaped and concatenated to intermediate feature maps within the generator's architecture. This allows the condition to influence the generation process at multiple levels of feature abstraction. For convolutional generators, $y$ might be spatially replicated to match the dimensions of the feature map before concatenation along the channel axis.

2. Conditional Modulation Techniques

More sophisticated methods involve using the condition $y$ to modulate the behavior of normalization layers within the generator.

• Conditional Batch Normalization (CBN): Standard Batch Normalization normalizes activations within a layer and then applies learned affine transformation parameters, scale ($\gamma$) and shift ($\beta$). In CBN, these parameters $\gamma$ and $\beta$ are no longer just learned globally but are instead predicted by small neural networks that take the condition $y$ as input. That is, $\gamma = f_\gamma(y)$ and $\beta = f_\beta(y)$. This allows the condition to dynamically control the feature statistics (mean and variance) throughout the generator, providing fine-grained control over the synthesis process, often influencing stylistic aspects related to $y$.

• Adaptive Instance Normalization (AdaIN): Popularized by StyleGAN, AdaIN is similar in spirit to CBN but operates on instance normalization. It aligns the mean and standard deviation of the content features (from $z$) to match the mean and standard deviation derived from the style input (which can be influenced by $y$ via a mapping network). This effectively "injects" the style or condition $y$ into the synthesis network at multiple points.

These modulation techniques often lead to better disentanglement and control compared to simple concatenation, especially for complex, high-resolution generation tasks.

Conditioning the Discriminator

The discriminator's role in a cGAN is twofold: determine if an input sample $x$ is real or fake, and verify if $x$ matches the given condition $y$. Without the second part, the generator might learn to ignore $y$ and produce realistic but irrelevant samples.

1. Input Concatenation

Similar to the generator, the most direct way to inform the discriminator about the condition is through concatenation.

• Concatenating Input and Condition: The input sample $x$ (either real or generated) is processed, often through early convolutional layers, to obtain a feature representation. The condition vector $y$ (potentially embedded) is also processed and reshaped. These two representations are then concatenated and fed into the later layers of the discriminator, which produce the final real/fake prediction.

Processing the input $x$ and condition $y$ separately before concatenating their representations for final discrimination.

2. Projection Discriminator

A more effective and widely adopted technique, especially for high-dimensional inputs and numerous classes, is the Projection Discriminator. Instead of simple concatenation, it explicitly incorporates the matching between the input $x$ and condition $y$ into the discriminator's output logit.

The architecture works as follows:

1. The condition $y$ (typically a class label) is embedded into a vector $v_y$.
2. The input image $x$ is passed through the main body of the discriminator network (e.g., convolutional layers) to obtain an intermediate feature representation $\phi(x)$.
3. The core idea is to calculate the dot product (or a similar bilinear pooling) between the image features $\phi(x)$ and the embedded condition $v_y$. This term, $\phi(x)^T v_y$, measures the compatibility or alignment between the image features and the specific condition.
4. The final output logit of the discriminator is computed as a sum of two terms: a term based solely on the image features (representing the "realness" independent of condition) and the projection term calculated in step 3.

$$\text{Output Logit} = W \phi(x) + \phi(x)^T v_y$$

Here, $W \phi(x)$ captures the unconditional real/fake score, while $\phi(x)^T v_y$ explicitly rewards the discriminator for recognizing images whose features $\phi(x)$ align well with the embedding $v_y$ of the correct condition $y$. This architecture strongly encourages the generator to produce samples that are not only realistic but also accurately reflect the given condition $y$.

Architecture of a Projection Discriminator, combining an unconditional score with a conditional projection term.

3. Auxiliary Classifier GANs (AC-GANs)

While distinct, AC-GANs share similarities. In an AC-GAN, the discriminator performs two tasks simultaneously: it predicts whether the input $x$ is real or fake, and it predicts the class label $y$ associated with $x$. The loss function includes both the standard adversarial loss and an auxiliary classification loss (e.g., cross-entropy). This forces the generator to produce samples that are not only realistic but also correctly classifiable by the discriminator according to the intended condition $y$. The architectural modification involves adding a separate output head to the discriminator for the class prediction task.

Choosing the right conditioning architecture depends on the nature of the condition $y$, the complexity of the generation task, and computational constraints. Concatenation is simpler to implement, while modulation techniques (like CBN) and projection discriminators often provide superior performance and control for complex conditional generation tasks.