Large Scale GAN Training (BigGAN)

"While architectures like ProGAN and StyleGAN focused on novel generator designs and progressive training, another significant direction in GAN development involved scaling. The goal was to use larger models and massive datasets (like ImageNet) to generate images with high fidelity and diversity, matching the complexity of photographs at higher resolutions. This pursuit led to the development of BigGAN."

Training GANs at a very large scale, however, presents unique challenges. Instability becomes a much more pronounced problem. Gradients can vanish or explode more easily, and the delicate balance between the generator and discriminator is harder to maintain across distributed training setups and enormous parameter counts. BigGAN introduced several techniques specifically designed to enable stable training under these demanding conditions.

Techniques for Large-Scale Training

BigGAN's success depends on a combination of architectural choices and stabilization methods tailored for scale:

1. Increased Model Capacity and Batch Size: BigGAN significantly increased the width (number of channels) and depth of both the generator and discriminator networks compared to prior models. Perhaps more importantly, it utilized substantially larger batch sizes during training (e.g., 2048 or more, distributed across many processors). Larger batches provide more stable and representative gradient estimates for both networks, which is particularly helpful in stabilizing the adversarial dynamics. This, however, necessitates significant computational resources (TPUs or large GPU clusters).

2. Self-Attention: As discussed previously, self-attention mechanisms allow the model to capture long-range dependencies across the image. BigGAN incorporates self-attention layers, particularly effective at higher resolutions where coordinating features across distant spatial locations becomes important for realism (e.g., ensuring the texture of fur is consistent across a generated animal's body).

3. Shared Embeddings and Conditional Batch Normalization: BigGAN excels at conditional image generation (producing images based on a class label, like "corgi" or "volcano"). It uses shared embeddings for class information that are linearly projected into the generator's batch normalization layers (Conditional Batch Normalization). This allows the class information to modulate the feature maps effectively throughout the generator without adding excessive parameters.

4. Hierarchical Latent Space: Instead of feeding the entire latent vector $z$ only to the initial layer, BigGAN splits $z$ and feeds chunks into multiple layers of the generator. This gives the model finer control over features at different levels of abstraction and resolution, contributing to both diversity and fidelity.

5. Orthogonal Regularization: This is a critical stabilization technique applied primarily to the generator's weights. The intuition is to encourage the weight matrices $W$ to be (or be close to) orthogonal, meaning $W^T W = I$. Orthogonal transformations preserve vector norms, which helps prevent gradient explosion or vanishing during backpropagation through deep networks. BigGAN applies a regularization term that penalizes weights for deviating from orthogonality:

   $$R_{\beta}(W) = \beta ||W^T W - I||_F^2$$

   Here, $||.||_F^2$ is the squared Frobenius norm, and $\beta$ is a small hyperparameter controlling the strength of the regularization. This simple constraint proved effective in stabilizing training for very deep generators.

6. Spectral Normalization (in Discriminator): While orthogonal regularization helps the generator, BigGAN (like many advanced GANs) uses Spectral Normalization in the discriminator. This technique, discussed further in Chapter 3, controls the Lipschitz constant of the discriminator by normalizing its weight matrices based on their largest singular value. This helps prevent the discriminator's gradients from becoming too large, contributing significantly to overall training stability.

The Truncation Trick: Trading Diversity for Fidelity

Even with these stabilization techniques, the samples generated from the full prior distribution $z \sim \mathcal{N}(0, I)$ might sometimes contain lower-quality or atypical examples residing in the tails of the distribution. BigGAN introduced the "truncation trick" as a post-hoc sampling method to improve the fidelity (visual quality and realism) of individual samples, albeit at the expense of overall sample diversity.

Instead of sampling $z$ directly from the standard normal distribution, components of $z$ are sampled from $\mathcal{N}(0, I)$ but are rejected and resampled if their magnitude $|z_i|$ exceeds a chosen threshold $\psi$. A smaller $\psi$ means tighter truncation, leading to samples closer to the "average" or high-density regions of the learned distribution, generally resulting in higher visual fidelity but less variety. Setting $\psi = 0$ would theoretically yield the "average" image, while a very large $\psi$ (or no truncation) uses the full latent space, maximizing diversity but potentially including less plausible outputs.

Illustration of the truncation trick. Samples $z$ are drawn from the full latent distribution $\mathcal{N}(0, I)$. BigGAN applies a threshold $\psi$ to filter out values from the low-probability tails ("Low Prob.") of this distribution. Only samples within the central "Truncated Region" (highlighted by the blue curve and labeled "High Fid.") are retained and fed to the generator. This improves image fidelity by avoiding outliers, but at the cost of reduced diversity in the generated outputs.

Impact and Computational Cost

BigGAN demonstrated that with sufficient scale and appropriate stabilization, GANs could produce diverse, high-resolution (e.g., 256x256, 512x512) images with remarkable fidelity, often difficult to distinguish from real photographs in ImageNet categories. It significantly raised the bar for image generation quality.

However, this comes at a substantial computational cost. Training BigGAN requires hundreds of TPU or GPU cores and days or weeks of training time, making it inaccessible for researchers and practitioners without access to large-scale computing infrastructure. Despite the cost, the techniques pioneered in BigGAN (especially orthogonal regularization, the truncation trick, and insights into large-batch training) have influenced many subsequent generative models.

Understanding BigGAN highlights the interaction between model scale, dataset size, architectural design, and stabilization techniques required to advance generative modeling. It serves as a case study in addressing the engineering and theoretical challenges that arise when scaling deep generative models.