ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Advanced GAN techniques
Chapter 1: GAN Foundations Revisited
The Generator-Discriminator Architecture
The Minimax Objective Function
Common Training Instabilities
Limitations of Vanilla GANs
Deep Convolutional GANs (DCGANs) Refresher
Chapter 2: Advanced GAN Architectures
Progressive Growing of GANs (ProGAN)
Style-Based Generator Architecture (StyleGAN)
StyleGAN2 Enhancements
Large Scale GAN Training (BigGAN)
Self-Attention Mechanisms in GANs
Unpaired Image-to-Image Translation (CycleGAN)
Implementing StyleGAN Components: Hands-on Practical
Chapter 3: GAN Training Dynamics and Stabilization
The Challenge of Non-Convergence
Mode Collapse: Causes and Consequences
Alternative Divergences: Wasserstein Distance
Weight Clipping in WGAN
Gradient Penalty (WGAN-GP)
Spectral Normalization
Two Time-Scale Update Rule (TTUR)
Relativistic GANs
Implementing WGAN-GP: Practice
Chapter 4: Conditional and Controllable Generation
Introduction to Conditional GANs (cGANs)
Architectures for cGANs
Information Maximizing GANs (InfoGAN)
StackGAN: Text-to-Image Synthesis
Controlling Attributes via Latent Space Manipulation
Disentanglement Metrics and Challenges
Building a Conditional GAN: Hands-on Practical
Chapter 5: Quantitative and Qualitative Evaluation of GANs
Challenges in Evaluating Generative Models
Qualitative Assessment: Visual Turing Tests
Inception Score (IS): Formulation and Limitations
Fréchet Inception Distance (FID): Formulation
Interpreting FID Scores
Precision and Recall for Distributions
Perceptual Path Length (PPL)
Calculating FID Score: Practice
Chapter 6: GANs Beyond Standard Image Generation
Challenges with Discrete Data: Text Generation
Reinforcement Learning Approaches (SeqGAN, RankGAN)
Continuous Approximations (Gumbel-Softmax)
Audio Synthesis with GANs (WaveGAN, SpecGAN)
Video Generation and Prediction
3D Data Generation (Point Clouds, Meshes)
Graph Generation with GANs
Chapter 7: Implementation, Optimization, and Tooling
Choosing Deep Learning Frameworks
Advanced Optimizers (AdamW, Lookahead)
Hyperparameter Tuning Strategies
Weight Initialization Techniques
Debugging Unstable GAN Training
Mixed Precision Training
Distributed Training Strategies for Large GANs
Profiling and Performance Optimization
Optimizing a GAN Implementation: Practice

Alternative Divergences: Wasserstein Distance

Vanilla GAN training often struggles with instability. The generator might produce only a narrow range of outputs (mode collapse), or the learning process might diverge entirely due to vanishing or exploding gradients. A significant contributing factor to these issues lies in the very nature of the objective function used: the Jensen-Shannon (JS) divergence.

While the JS divergence is a valid way to measure the difference between probability distributions, it has properties that make it less than ideal for the adversarial training dynamics of GANs, especially in high-dimensional spaces like images.

The Problem with Jensen-Shannon Divergence

The original GAN paper showed that minimizing the GAN objective corresponds to minimizing the JS divergence between the real data distribution PrP_rPr​ and the generated data distribution PgP_gPg​. However, what happens when PrP_rPr​ and PgP_gPg​ have supports that do not overlap significantly, or when they lie on low-dimensional manifolds within a much higher-dimensional space (a common scenario for image data)?

In such cases, it's often possible to find a discriminator that perfectly separates real samples from generated samples. When the discriminator achieves near-perfect accuracy:

  1. The JS divergence between PrP_rPr​ and PgP_gPg​ effectively saturates at its maximum value (log 2).
  2. More importantly, the gradients of the loss function with respect to the generator's parameters approach zero.

Think about it: if the discriminator can always tell real from fake, it provides very little useful information back to the generator about how to improve. The generator essentially hits a wall, receiving no meaningful signal to guide its updates. This vanishing gradient problem is a primary cause of training failure and mode collapse, as the generator stops learning effectively.

We need a way to measure the distance between PrP_rPr​ and PgP_gPg​ that provides smoother, more informative gradients even when the distributions are quite different or don't overlap perfectly.

Introducing the Wasserstein Distance (Earth Mover's Distance)

Enter the Wasserstein-1 distance, often referred to as the Earth Mover's Distance (EMD). Imagine PrP_rPr​ and PgP_gPg​ as two different piles of dirt (probability mass). The EMD represents the minimum "cost" required to transform one pile into the other. The cost is defined as the amount of dirt moved multiplied by the distance it is moved.

The Wasserstein distance calculates the minimum cost to transport mass from the generated distribution PgP_gPg​ to match the real distribution PrP_rPr​.

Crucially, even if the two piles of dirt (distributions) are located far apart (disjoint supports), the cost of moving the dirt provides a meaningful measure of how different they are. Unlike JS divergence, the Wasserstein distance doesn't saturate abruptly and generally provides a smoother metric.

The Kantorovich-Rubinstein Duality

Calculating the EMD directly using its primal definition involves finding an optimal "transport plan" (how much dirt to move from each point in PgP_gPg​ to each point in PrP_rPr​), which is computationally intractable in most interesting cases. Fortunately, the Kantorovich-Rubinstein duality provides an alternative formulation for the Wasserstein-1 distance:

W1(Pr,Pg)=sup⁡∥f∥L≤1(Ex∼Pr[f(x)]−Ex~∼Pg[f(x~)])W_1(P_r, P_g) = \sup_{\|f\|_L \le 1} \left( \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{\tilde{x} \sim P_g}[f(\tilde{x})] \right)W1​(Pr​,Pg​)=∥f∥L​≤1sup​(Ex∼Pr​​[f(x)]−Ex~∼Pg​​[f(x~)])

Let's break this down:

  • W1(Pr,Pg)W_1(P_r, P_g)W1​(Pr​,Pg​) is the Wasserstein-1 distance between the real distribution PrP_rPr​ and the generated distribution PgP_gPg​.
  • sup⁡\supsup denotes the supremum, which is the least upper bound (similar to a maximum).
  • ∥f∥L≤1\|f\|_L \le 1∥f∥L​≤1 means the supremum is taken over all functions fff that are 1-Lipschitz continuous. A function fff is K-Lipschitz if ∣f(x1)−f(x2)∣≤K∥x1−x2∥|f(x_1) - f(x_2)| \le K \|x_1 - x_2\|∣f(x1​)−f(x2​)∣≤K∥x1​−x2​∥ for all x1,x2x_1, x_2x1​,x2​ in its domain. Essentially, a 1-Lipschitz function's rate of change is bounded by 1. This constraint is fundamental.
  • Ex∼Pr[f(x)]\mathbb{E}_{x \sim P_r}[f(x)]Ex∼Pr​​[f(x)] is the expected value of f(x)f(x)f(x) when xxx is sampled from the real data distribution PrP_rPr​.
  • Ex~∼Pg[f(x~)]\mathbb{E}_{\tilde{x} \sim P_g}[f(\tilde{x})]Ex~∼Pg​​[f(x~)] is the expected value of f(x~)f(\tilde{x})f(x~) when x~\tilde{x}x~ is sampled from the generator's distribution PgP_gPg​ (where x~=g(z)\tilde{x} = g(z)x~=g(z), with zzz sampled from some prior noise distribution p(z)p(z)p(z)).

This dual formulation states that the Wasserstein distance is the maximum possible difference in expectations achievable by evaluating a 1-Lipschitz function fff on samples from the two distributions.

Wasserstein GANs (WGANs)

The insight of the WGAN paper (Arjovsky et al., 2017) was to use this dual formulation to define a new GAN objective. The core idea is:

  1. Replace the Discriminator with a Critic: Instead of a discriminator outputting the probability of a sample being real (a value between 0 and 1, typically via a sigmoid), we use a network, called the critic (let's denote it fwf_wfw​ with parameters www), that outputs an unbounded real number.
  2. Train the Critic to Approximate the Supremum: The critic fwf_wfw​ is trained to make the term (Ex∼Pr[fw(x)]−Ex~∼Pg[fw(x~)])(\mathbb{E}_{x \sim P_r}[f_w(x)] - \mathbb{E}_{\tilde{x} \sim P_g}[f_w(\tilde{x})])(Ex∼Pr​​[fw​(x)]−Ex~∼Pg​​[fw​(x~)]) as large as possible. By doing so, under the right conditions (specifically, if fwf_wfw​ remains within the space of 1-Lipschitz functions), the critic's objective value approximates the Wasserstein distance W1(Pr,Pg)W_1(P_r, P_g)W1​(Pr​,Pg​).
  3. Train the Generator to Minimize the Critic's Output: The generator gθg_\thetagθ​ (with parameters θ\thetaθ) is trained to produce samples x~=gθ(z)\tilde{x} = g_\theta(z)x~=gθ​(z) that make the critic's output fw(x~)f_w(\tilde{x})fw​(x~) larger (closer to the output for real samples), thereby minimizing the difference the critic is trying to maximize. This effectively minimizes the estimated Wasserstein distance.

The objectives become:

  • Critic Loss: Maximize LCritic=Ex∼Pr[fw(x)]−Ez∼p(z)[fw(gθ(z))]L_{\text{Critic}} = \mathbb{E}_{x \sim P_r}[f_w(x)] - \mathbb{E}_{z \sim p(z)}[f_w(g_\theta(z))]LCritic​=Ex∼Pr​​[fw​(x)]−Ez∼p(z)​[fw​(gθ​(z))]. In practice, neural networks are trained via gradient descent (minimization), so we typically minimize −LCritic-L_{\text{Critic}}−LCritic​.
  • Generator Loss: Minimize LGenerator=−Ez∼p(z)[fw(gθ(z))]L_{\text{Generator}} = - \mathbb{E}_{z \sim p(z)}[f_w(g_\theta(z))]LGenerator​=−Ez∼p(z)​[fw​(gθ​(z))]. The generator tries to fool the critic by producing samples that score highly (as if they were real).

Why This Helps Stability

Using the Wasserstein distance via the critic provides several advantages:

  1. Meaningful Gradients: The Wasserstein distance provides gradients that are non-zero and informative even when PrP_rPr​ and PgP_gPg​ have disjoint supports. This drastically reduces the vanishing gradient problem that plagued original GANs.
  2. Improved Correlation with Sample Quality: Empirically, the WGAN loss (the estimated Wasserstein distance) often correlates much better with the perceptual quality of the generated samples during training. Lower loss generally means better-looking images, which wasn't always true for the original GAN loss.
  3. Reduced Risk of Mode Collapse: Theoretical arguments and empirical results suggest that WGANs are less prone to mode collapse, likely due to the more stable gradients allowing the generator to explore the data distribution more effectively.

The Lipschitz Constraint: A Practical Hurdle

The theory behind WGAN relies heavily on the critic fwf_wfw​ being (or approximating) a 1-Lipschitz function. If the critic's weights grow too large, it can violate this constraint, making the approximation of the Wasserstein distance invalid and potentially leading back to unstable training.

Therefore, an important part of implementing WGANs is enforcing this Lipschitz constraint on the critic network. The original WGAN paper proposed a simple, albeit sometimes problematic, method called weight clipping. Subsequent research introduced more sophisticated and often preferred techniques like the gradient penalty (WGAN-GP) and spectral normalization. We will examine these methods for enforcing the Lipschitz constraint in the following sections.

  • Improved Training of Wasserstein GANs, Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, Aaron Courville, 2017 Advances in Neural Information Processing Systems 30 (NIPS 2017) DOI: 10.48550/arXiv.1704.00028 - Proposes gradient penalty as a more effective and stable alternative to weight clipping for enforcing the Lipschitz constraint in WGANs.
  • Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - Provides a comprehensive introduction to deep learning, including foundational explanations of GANs, training issues, and different divergence metrics.

© 2026 ApX Machine Learning