StyleGAN2 Enhancements

While the original StyleGAN architecture represented a significant leap forward in generating high-resolution, visually appealing images with disentangled controls, practitioners quickly identified certain characteristic visual artifacts and limitations. StyleGAN2 was developed specifically to address these shortcomings, refining the generator and training process for improved fidelity and smoother results.

Addressing Normalization Artifacts: Modulated Demodulation

One of the most noticeable issues in StyleGAN outputs was the appearance of blob-like or "water droplet" artifacts. These were traced back to the Adaptive Instance Normalization (AdaIN) operation used within the synthesis network. Recall that AdaIN first normalizes the feature map activations (zero mean, unit variance) within each channel, effectively erasing magnitude information, and then applies learned scale and bias parameters derived from the style vector $w$.

$$\text{AdaIN}(x_i, y) = y_{s,i} \frac{x_i - \mu(x_i)}{\sigma(x_i)} + y_{b,i}$$

The hypothesis was that the network struggled to reintroduce magnitude information solely through the learned bias $y_{b,i}$ after normalization had removed it. This sometimes led to the generator creating excessively strong, localized signals (the "droplets") to compensate.

StyleGAN2 replaces AdaIN with a technique called Modulated Convolution followed by Demodulation (ModDemod).

1. Modulation: Instead of applying style information after normalization, StyleGAN2 incorporates the style directly into the convolutional weights before the convolution. For a convolutional layer with weights $w_{ijk}$ (output channel $j$, input channel $i$, spatial kernel position $k$) and an incoming style scale $s_i$ derived from $w$ for input channel $i$, the modulated weights $w'_{ijk}$ become:

   $$w'_{ijk} = s_i \cdot w_{ijk}$$

   This scales the influence of each input feature map based on the style.

2. Demodulation: Applying modulated weights can drastically change the overall scale of the output activations. To counteract this and prevent signal escalation or vanishing, demodulation is applied after the modulated convolution. It normalizes the output feature map $x'$ based on the L2 norm of the modulated weights used to produce it. For each output feature map $j$:

   $$x''_j = x'_j / \sqrt{\sum_{i,k} (w'_{ijk})^2 + \epsilon}$$

   Here, $\epsilon$ is a small constant for numerical stability. This demodulation step ensures that the standard deviation of the output activations remains roughly unit, effectively standardizing the signal magnitude based on the learned weights themselves, rather than erasing information via instance normalization.

This combined ModDemod operation removes the need for AdaIN and, consequently, eliminates the primary source of the droplet artifacts, leading to cleaner feature maps.

Eliminating Phase Artifacts: Architectural Changes

Another artifact observed in StyleGAN was related to phase. During interpolations in the latent space (or when generating animations), certain features like eyes or teeth might appear "stuck" to the image canvas instead of moving naturally with the apparent pose or viewpoint change. This suggested issues with how the network represented spatial frequencies and transformations.

The StyleGAN2 authors identified the progressive growing technique (inherited from ProGAN) as a potential contributor. While progressive growing helps stabilize training for high resolutions initially, dynamically changing the network architecture during training might interfere with the generator learning consistent phase behavior across different feature scales.

StyleGAN2 abandons progressive growing. Instead, it trains the full-resolution network from the beginning. To manage the flow of information from lower to higher resolutions effectively (which progressive growing aimed to help with), StyleGAN2 employs:

• Residual Connections: Similar to ResNets, skip connections are used within the generator architecture.
• Multi-Scale Outputs: Inspired by MSG-GAN, the architecture uses multiple toRGB layers that map intermediate feature resolutions directly to RGB outputs. These intermediate RGB outputs are then upsampled and summed together to form the final image. This allows gradients to flow more directly to earlier layers and encourages the network to utilize features from all resolution levels simultaneously.

Simplified view of StyleGAN2 generator architecture, highlighting skip connections (toRGB outputs from multiple resolutions) which replace progressive growing. Intermediate outputs are combined to form the final image.

Removing progressive growing and using this alternative multi-scale architecture design significantly reduced the phase artifacts, leading to more natural-looking transformations and interpolations.

Improving Latent Space Smoothness: Path Length Regularization

To further improve image quality and the smoothness of the latent space $W$, StyleGAN2 introduces path length regularization ($L_{path}$) during training. The goal is to encourage the mapping from $W$ to the generated image to be more regular, meaning that a step of a fixed size in $W$ should correspond to a roughly fixed-magnitude change in the image, regardless of the location in $W$ or the direction of the step. Abrupt changes in the image for small steps in $W$ are undesirable and correlate with lower visual quality.

The path length regularizer penalizes the deviation of the magnitude of the image space gradient (Jacobian) from a constant value. It's formulated as:

$$L_{path} = \mathbb{E}_{w, y \sim N(0,I)} \left[ \left( \| J_w^T y \|_2 - a \right)^2 \right]$$

Where:

• $w$ is a latent code sampled from $W$.
• $G(w)$ is the generated image.
• $J_w = \nabla_w G(w)$ is the Jacobian matrix of the generator output with respect to the latent code $w$.
• $y$ is a random image with unit-variance normally distributed pixel values (used to project the gradient into a random image-space direction, avoiding the need to compute the full, potentially huge, Jacobian).
• $\| J_w^T y \|_2$ represents the magnitude of the change in the image resulting from steps in $w$.
• $a$ is a dynamically updated moving average of the $\| J_w^T y \|_2$ values, acting as the target constant magnitude.

By minimizing this loss, the generator is incentivized to make the mapping $G: W \rightarrow \text{Image}$ less "curvy" and more predictable locally. This regularization improves model conditioning, enhances image quality (often measured by FID), and leads to perceptually smoother interpolations.

Reducing Computational Cost: Lazy Regularization

Regularization terms like $L_{path}$ or the R1 gradient penalty for the discriminator can add significant computational overhead to each training iteration. StyleGAN2 introduces the concept of "lazy regularization," where these computationally intensive regularization terms are computed and applied less frequently than the main generator and discriminator losses. For instance, the path length regularization might only be calculated once every 16 mini-batches, while the main adversarial loss is computed every mini-batch. This simple trick drastically reduces the training time cost associated with these regularizers without substantially impacting their effectiveness or the final model quality.

Summary of StyleGAN2 Enhancements

Collectively, these refinements in StyleGAN2 led to substantial improvements:

• Reduced Artifacts: Elimination of droplet artifacts (via ModDemod) and phase artifacts (via architectural changes).
• Improved Metrics: State-of-the-art FID scores and better PPL metrics compared to the original StyleGAN upon its release.
• Higher Visual Fidelity: Generated images often exhibit finer details and greater realism.
• Smoother Interpolations: Latent space traversals produce more coherent and perceptually smooth changes in the generated images.
• More Efficient Training: Lazy regularization allows complex regularization schemes without prohibitive computational costs.

These enhancements solidified StyleGAN2 as a benchmark architecture for high-resolution image synthesis and further demonstrated the importance of careful architectural design and training regularization in achieving top-tier generative results. Understanding these improvements provides valuable insights into tackling common challenges in GAN development.