GAN Fundamentals

Generative Adversarial Networks (GANs), introduced by Ian Goodfellow and colleagues in 2014, represent a powerful framework for learning generative models. As outlined previously, GANs employ an adversarial process involving two neural networks: a Generator ($G$) and a Discriminator ($D$). Let's examine their roles and interactions more closely.

The Generator Network (G)

The Generator's task is to synthesize data that appears indistinguishable from real data. It takes a random noise vector $z$ as input, typically sampled from a simple prior distribution $p_z(z)$ like a multivariate Gaussian or uniform distribution. This noise vector $z$ resides in a lower-dimensional latent space. The Generator acts as a mapping function, transforming this latent vector into a high-dimensional data sample $G(z)$ in the data space (e.g., an image).

$G: \mathcal{Z} \rightarrow \mathcal{X}$

Here, $\mathcal{Z}$ is the latent space and $\mathcal{X}$ is the data space. The goal for $G$ is to learn a distribution $p_g$ over $\mathcal{X}$ that matches the true data distribution $p_{data}$. Architecturally, for tasks like image generation, the Generator often uses layers like transposed convolutions (sometimes called deconvolutions) to upsample the low-dimensional input noise into a full-sized image. Its objective during training is purely to produce outputs $G(z)$ that the Discriminator classifies as real.

The Discriminator Network (D)

The Discriminator acts as a binary classifier. Its input is a data sample $x$ (which could be either a real sample from $p_{data}$ or a generated sample $G(z)$ from $p_g$), and its output $D(x)$ is a single scalar representing the probability that $x$ came from the real data distribution $p_{data}$.

$D: \mathcal{X} \rightarrow [0, 1]$

Ideally, $D(x)$ should be close to 1 for real samples and close to 0 for generated samples. For image data, the Discriminator is commonly implemented as a standard Convolutional Neural Network (CNN) that outputs a probability. Its objective during training is to correctly distinguish between real and generated samples.

The Adversarial Min-Max Game

The training process pits $G$ and $D$ against each other in a zero-sum game. The core of this interaction is captured by the value function $V(D, G)$ introduced earlier:

$\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{data}(x)}[\log D(x)] + \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))]$

Let's break this down:

1. Maximizing $D$: The Discriminator $D$ wants to maximize $V(D, G)$. It does this by:

   • Assigning high probabilities (close to 1) to real samples $x \sim p_{data}(x)$, maximizing the first term $\mathbb{E}_{x \sim p_{data}(x)}[\log D(x)]$.
   • Assigning low probabilities (close to 0) to generated samples $G(z)$, which maximizes the second term $\mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))]$ because $1 - D(G(z))$ will be close to 1. Essentially, $D$ learns to classify real vs. fake accurately by maximizing the standard binary cross-entropy loss for classification.

2. Minimizing $G$: The Generator $G$ wants to minimize $V(D, G)$ by making $D$ perform poorly on generated samples. Since $G$ only affects the second term, it tries to make $D(G(z))$ as close to 1 as possible (fooling the discriminator). This minimizes $\log(1 - D(G(z)))$, which tends towards $-\infty$ as $D(G(z)) \to 1$.

This min-max formulation establishes an equilibrium point. Theoretically, if both $G$ and $D$ have sufficient capacity and the training process converges optimally, the generator's distribution $p_g$ will perfectly match the real data distribution $p_{data}$. At this point, the discriminator cannot distinguish real from generated samples better than chance, meaning $D(x) = 0.5$ for all $x$. The value function $V(D, G)$ converges to $-\log 4$.

Basic architecture of a Generative Adversarial Network showing the flow of random noise and real data through the Generator and Discriminator.

Training Dynamics and Implementation Details

In practice, training $G$ and $D$ simultaneously using standard gradient descent is unstable. Instead, training alternates between updating $D$ and $G$:

1. Update Discriminator: Sample a minibatch of noise vectors $\{z^{(1)}, ..., z^{(m)}\}$ and a minibatch of real data examples $\{x^{(1)}, ..., x^{(m)}\}$. Update the parameters of $D$ by ascending the stochastic gradient of $V(D, G)$: $\nabla_{\theta_d} \frac{1}{m} \sum_{i=1}^m [\log D(x^{(i)}) + \log(1 - D(G(z^{(i)})))]$ This step might be repeated for $k$ iterations to ensure $D$ remains effective.

2. Update Generator: Sample a minibatch of noise vectors $\{z^{(1)}, ..., z^{(m)}\}$. Update the parameters of $G$ by descending the stochastic gradient of $V(D, G)$, specifically targeting the second term: $\nabla_{\theta_g} \frac{1}{m} \sum_{i=1}^m \log(1 - D(G(z^{(i)})))$

A significant practical issue arises with the generator's loss function $\log(1 - D(G(z)))$. When the discriminator becomes very effective early in training, it correctly assigns $D(G(z)) \approx 0$ to generated samples. In this region, the gradient of $\log(1 - x)$ with respect to $x$ is very small (saturates), providing weak learning signals for the generator.

To counteract this saturation, a common modification is to change the generator's objective from minimizing $\mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))]$ to maximizing $\mathbb{E}_{z \sim p_z(z)}[\log D(G(z))]$. This is often referred to as the "non-saturating" heuristic objective. While it doesn't represent the original min-max game exactly, it aims for the same goal (making $D(G(z))$ close to 1) but provides much stronger gradients early in training when $D(G(z))$ is small. In practice, this means updating $G$ by ascending the gradient: $\nabla_{\theta_g} \frac{1}{m} \sum_{i=1}^m \log D(G(z^{(i)}))$

While the fundamental GAN framework is elegant, achieving stable training and high-quality results requires careful consideration of architecture choices, loss function variants, and optimization strategies. Difficulties like training instability (oscillations or divergence) and mode collapse (where the generator produces only a limited variety of samples) are common. These challenges motivate the advanced techniques and architectures explored in subsequent chapters.