Progressive Growing of GANs (ProGAN)

Training Generative Adversarial Networks directly on high-resolution images presents significant challenges. Large networks are difficult to optimize, gradients can vanish or explode, and the generator and discriminator may struggle to coordinate their learning process effectively, especially in the early stages when the generated images bear little resemblance to the target distribution. Generating fine details while simultaneously learning the coarse structure of an image from scratch is demanding.

Progressive Growing of GANs (ProGAN), introduced by Karras et al. (NVIDIA) in 2017, offers an elegant solution to this problem. Instead of training a single large network for the target high resolution from the beginning, ProGAN starts with very low-resolution images (e.g., 4x4 pixels) and incrementally adds layers to both the generator (G) and discriminator (D) to handle progressively higher resolutions (8x8, 16x16, ..., up to 1024x1024 or higher).

The Core Idea: Grow Incrementally

The fundamental principle is to first train the networks to understand the coarse structure of the image distribution at a low resolution. Once this initial stage converges reasonably well, new layers are added to both G and D to double the spatial resolution. The previously trained layers provide a stable foundation, and the new layers focus on learning the finer details specific to the increased resolution. This process repeats until the desired output resolution is reached.

Stabilizing Growth with Layer Fading

A sudden introduction of new layers can shock the system and destabilize training. ProGAN addresses this by smoothly fading in the new layers. When transitioning from resolution $R \times R$ to $2R \times 2R$, the new layers are added, but their influence is gradually increased using a parameter $\alpha$ that ramps up from 0 to 1 over many iterations.

Here's the generator:

1. Existing layers generate an image at resolution $R \times R$. This output is upsampled to $2R \times 2R$.
2. The newly added layers also process the feature maps from the previous stage and produce an image block operating at the new $2R \times 2R$ resolution.
3. The final output is a weighted combination of these two paths: $\text{Output}_{2R \times 2R} = (1 - \alpha) \times (\text{Upsampled Output}_{R \times R}) + \alpha \times (\text{New Layer Output}_{2R \times 2R})$

A similar fading process occurs in the discriminator, but in reverse: input images at $2R \times 2R$ are processed by the new layers, while a downsampled version ($R \times R$) bypasses them and goes into the older part of the network. The discriminator's decision is based on a convex combination (controlled by $\alpha$) of the outputs from both paths.

Progressive growing phase transition. New layers (blue in G, red in D) are added to handle resolution $2R \times 2R$. Their output is combined with the output from the previous $R \times R$ stage (upsampled in G, downsampled in D) using a parameter $\alpha$ that increases from 0 to 1, ensuring a smooth transition.

This gradual adaptation allows the network to incorporate the new capacity for detail without disrupting the already learned stable features from lower resolutions.

Benefits of Progressive Training

1. Training Stability: By focusing on simpler, low-resolution structures first, the optimization problem is initially easier. The network establishes a strong foundation before tackling complex high-frequency details. This significantly reduces the likelihood of catastrophic mode collapse or divergent training dynamics often seen when training large GANs from scratch.
2. Faster Training: Although training involves multiple stages, each stage trains faster than attempting to train the full high-resolution network from the outset. The early stages converge quickly on the low-resolution data.
3. High-Resolution Synthesis: ProGAN was groundbreaking in its ability to generate high-quality, coherent images at resolutions like 1024x1024, a significant leap at the time.

Architectural Techniques and Supporting Approaches

While progressive growing is the central idea, the success of ProGAN also relies on several other architectural choices and training techniques applied at each stage:

• Equalized Learning Rate: Instead of relying on careful weight initialization, ProGAN uses a runtime mechanism to dynamically scale the weights in each layer. Specifically, weights $w_i$ are scaled by a per-layer constant $c$ derived from He's initializer: $\hat{w}_i = w_i / c$. This ensures that the variance of outputs remains consistent across layers and magnitudes, improving stability.
• Pixelwise Feature Vector Normalization: Applied after each convolutional layer in the generator, this technique normalizes the feature vector at each pixel to unit length. This helps prevent escalating signal magnitudes, particularly useful in generator networks.
• Minibatch Standard Deviation: To encourage the generator to produce more diverse samples and prevent mode collapse, a layer is added near the end of the discriminator. This layer computes the standard deviation of features across samples in the minibatch for each spatial location, calculates the average standard deviation across all features and locations, and appends this scalar value as an additional feature map to the input of the final discriminator layer. This gives the discriminator a signal about the batch statistics, implicitly encouraging the generator to create batches with statistics similar to real data.

ProGAN's Impact

Progressive Growing demonstrated a powerful methodology for training GANs for high-resolution outputs. It highlighted the importance of curriculum learning principles (starting simple, gradually increasing complexity) in the context of generative models. While architectures like StyleGAN have built upon and refined these ideas, the core concept of progressive resolution increase introduced by ProGAN remains an important technique in the GAN practitioner's toolkit, showcasing how careful architectural design can overcome fundamental training hurdles.