Progressive Growing of GANs (ProGAN)

Generating high-resolution images like 1024x1024 pixels directly with a standard Generative Adversarial Network (GAN) poses significant training challenges. As network depth and image dimensions increase, gradients can become unstable, leading to divergence or slow convergence. Furthermore, the generator might struggle to learn coarse structure and fine details simultaneously from the beginning. The Progressive Growing of GANs (ProGAN) approach, introduced by Karras et al. (NVIDIA), offers an elegant solution by incrementally increasing the complexity of the task during training.

The main idea behind ProGAN is to start training both the generator (G) and the discriminator (D) with very low-resolution images (e.g., 4x4 pixels) and then progressively add new layers to both networks to handle higher resolutions (8x8, 16x16, ..., up to 1024x1024 or higher). This incremental approach allows the network to first learn the large-scale structure of the image distribution at low resolutions and then shift its focus towards finer details as the resolution increases.

The Growth Process

Training begins with a simple generator and discriminator pair operating on low-resolution images (e.g., 4x4). Once this initial network pair shows signs of convergence and stability, new layers are added to both G and D to double the image resolution (e.g., move from 4x4 to 8x8).

Critically, this transition is not abrupt. The newly added layers are smoothly "faded in" over a period of training iterations. This is achieved using a parameter $\alpha$, which ranges from 0 to 1. When a new block of layers (handling the higher resolution) is added:

1. In the Generator: The output from the previous resolution block is upscaled, and the output from the new block (also taking the upscaled features as input) is computed. The final output is a weighted combination: (\text{output}_{\text{new}}) \times \alpha + (\text{upscaled_output}_{\text{old}}) \times (1 - \alpha)
2. In the Discriminator: The input image is processed by the new layers, and the result is downscaled. The input image is also downscaled directly and processed by the previous resolution block. The discriminator uses a weighted combination of these two paths: (\text{from_new_layers}) \times \alpha + (\text{from_old_layers_after_downscale}) \times (1 - \alpha)

The parameter $\alpha$ is gradually increased from 0 to 1. Initially ($\alpha=0$), the new layers have no effect, and the network operates as it did at the previous resolution. As $\alpha$ increases, the contribution of the new layers grows, allowing the network to adapt smoothly to the higher resolution task. Once $\alpha=1$, the old connection paths are effectively removed, and the network fully operates at the new, higher resolution. This process repeats for subsequent resolution increases.

Diagram illustrating the ProGAN training stages. Initially (Stage 1), G and D operate at low resolution (4x4). In Stage 2, new layers for 8x8 resolution are added and faded in using parameter $\alpha$. In Stage 3, the 8x8 network is trained stably after the fade-in is complete ($\alpha=1$). This process repeats for higher resolutions.

Supporting Techniques for Stability and Quality

ProGAN incorporates several additional techniques to further improve training stability and image quality, particularly at higher resolutions:

• Minibatch Standard Deviation: To combat mode collapse (where the generator produces only a limited variety of samples), a minibatch standard deviation layer is typically added towards the end of the discriminator. This layer calculates the standard deviation of features across all spatial locations and samples in the current minibatch. This statistic is then averaged and replicated into an additional feature map, which is concatenated with the original features. By providing the discriminator with information about the batch statistics, it can implicitly penalize the generator if it produces batches with unnaturally low variation, encouraging diversity.

• Equalized Learning Rate: Standard weight initialization schemes (like Xavier or He) don't always prevent gradients from exploding or vanishing in very deep networks, especially with varying activation functions or architectures. ProGAN uses equalized learning rate, a form of dynamic weight scaling. At runtime, before each forward or backward pass through a convolutional or fully connected layer, the weights $w_i$ are scaled by a factor $c$: $\hat{w}_i = w_i / c$ Here, $c$ is a per-layer normalization constant, typically calculated using He's initializer's principle: c = \sqrt{\frac{2}{\text{fan_in}}}, where \text{fan_in} is the number of input connections to the layer. This scaling ensures that the variance of the outputs (and gradients) remains approximately constant across layers, regardless of the number of connections or parameter scale, effectively equalizing the learning speed for all weights.

• Pixel Normalization: Applied within the generator after each convolutional layer (before the activation function). This technique normalizes the feature vector at each pixel $(x, y)$ to unit length: $b_{x,y} = \frac{a_{x,y}}{\sqrt{\frac{1}{N} \sum_{j=0}^{N-1} (a_{x,y}^j)^2 + \epsilon}}$ Here, $N$ is the number of feature channels, $a_{x,y}$ is the original feature vector at pixel $(x, y)$, $b_{x,y}$ is the normalized vector, and $\epsilon$ (e.g., $10^{-8}$) prevents division by zero. This local response normalization acts similarly to Batch Normalization but without relying on batch statistics. It prevents the signal magnitudes within the generator from spiraling out of control due to potential competition between the generator and discriminator, contributing significantly to training stability.

Advantages

The progressive growing approach offers several advantages:

• Stable High-Resolution Synthesis: It dramatically stabilizes the training process for high resolutions (1024x1024), which was previously very difficult.
• Faster Initial Training: Early stages focus on low-resolution images, requiring less computation and allowing the network to quickly learn coarse features.
• Improved Sample Quality: The gradual refinement process often leads to higher-fidelity generated images compared to training a large network from scratch.

However, there are considerations:

• Increased Training Time: While individual stages might be faster initially, the overall training time across all stages can be substantial.
• Architectural Complexity: Designing and implementing the smoothly fading layers adds complexity compared to a static architecture.
• Hyperparameter Sensitivity: The fade-in schedule ($\alpha$ progression) and the timing of resolution transitions require careful tuning.

ProGAN represented a significant step forward in generating high-resolution, high-quality images. Its core ideas of progressive training and specialized normalization techniques have influenced subsequent state-of-the-art models, including the StyleGAN family which we will examine next.