GAN-Based Vocoders (MelGAN, HiFi-GAN)

While autoregressive models like WaveNet generate high-fidelity audio sample by sample, and flow-based models like WaveGlow offer parallel synthesis through invertible transformations, Generative Adversarial Networks (GANs) present another powerful approach for waveform generation. GANs use a competitive training process between a generator and a discriminator network, proving highly effective at producing realistic-looking (and sounding) data. Adapting this approach for vocoding has led to models that offer an excellent trade-off between computational efficiency and audio quality.

The GAN Principle for Vocoding

In the context of vocoding, the GAN framework operates as follows:

1. Generator (G): Takes an intermediate acoustic representation, typically a mel-spectrogram, as input and attempts to generate a corresponding raw audio waveform. The generator's architecture is often designed for efficient upsampling, commonly using transposed convolutions.
2. Discriminator (D): Takes a raw audio waveform (either real or generated by G) as input and tries to classify it as real or fake. The discriminator learns to identify the subtle characteristics and potential artifacts present in synthesized audio.

The training involves an adversarial game: the generator tries to fool the discriminator by producing increasingly realistic waveforms, while the discriminator improves its ability to distinguish real from generated audio. This dynamic pushes the generator towards producing waveforms that are perceptually indistinguishable from real recordings.

Mathematically, this is often formulated as a minimax game with a value function $V(G, D)$. For instance, using the standard GAN loss:

$$\min_{G} \max_{D} V(G, D) = \mathbb{E}_{\mathbf{x} \sim p_{data}(\mathbf{x})} [\log D(\mathbf{x})] + \mathbb{E}_{\mathbf{z} \sim p_{z}(\mathbf{z})} [\log(1 - D(G(\mathbf{z})))]$$

Here, $\mathbf{x}$ represents real audio data, and $\mathbf{z}$ represents the input conditioning (mel-spectrograms) for the generator $G$. In practice, variations like least-squares GAN (LSGAN) loss are often used for more stable training.

MelGAN: Efficient Adversarial Waveform Generation

MelGAN was one of the first successful GAN-based vocoders demonstrating significantly faster inference than autoregressive models while maintaining good audio quality.

Architecture

• Generator: MelGAN's generator is typically a fully convolutional architecture using transposed convolutions to upsample the input mel-spectrogram temporally until it matches the desired audio waveform resolution. Residual blocks are often incorporated within the upsampling layers. Its non-autoregressive nature makes it computationally efficient during inference.
• Discriminator: An important innovation in MelGAN is the use of a multi-scale discriminator. Instead of a single discriminator analyzing the full-resolution audio, MelGAN employs multiple discriminators operating on downsampled versions of the input waveform (e.g., original, downsampled by 2, downsampled by 4). Each discriminator has a similar convolutional architecture but looks at the waveform at a different resolution. This encourages the generator to produce audio that is realistic across different time scales, capturing both fine-grained details and overall structure.

Simplified architecture of MelGAN, showing the generator producing a waveform from a mel-spectrogram and the multi-scale discriminator evaluating its realism at different resolutions.

Loss Function

MelGAN's training objective combines the adversarial loss (encouraging realistic outputs) with a feature matching loss. The feature matching loss helps stabilize training by penalizing discrepancies between the intermediate feature map activations within the discriminators for real and generated samples. This acts as a perceptual guide for the generator, beyond just fooling the final classification layer of the discriminator.

$$L_G = \sum_{k=1}^K \mathbb{E}_{(\mathbf{x}, \mathbf{z})} [ || D_k(G(\mathbf{z})) - D_k(\mathbf{x}) ||_1 ] \quad \text{(Feature Matching Loss)}$$ $$L_{Adv}(G, D) = \sum_{k=1}^K \mathbb{E}_{\mathbf{z}} [ (D_k(G(\mathbf{z})) - 1)^2 ] \quad \text{(Generator Adversarial Loss - LSGAN variant)}$$ $$L_{Adv}(D, G) = \sum_{k=1}^K \mathbb{E}_{\mathbf{x}} [ (D_k(\mathbf{x}) - 1)^2 ] + \mathbb{E}_{\mathbf{z}} [ (D_k(G(\mathbf{z})))^2 ] \quad \text{(Discriminator Adversarial Loss - LSGAN variant)}$$

The total generator loss is a weighted sum of $L_G$ and $L_{Adv}(G, D)$.

MelGAN provides a substantial speed-up over autoregressive models, making it practical for near real-time synthesis, although the output quality might sometimes contain minor artifacts compared to the best autoregressive or flow-based models.

HiFi-GAN: High-Fidelity Generative Adversarial Networks

HiFi-GAN builds upon the success of MelGAN, aiming for higher audio fidelity and improved perceptual quality while retaining computational efficiency. It introduces refinements to both the generator and discriminator architectures.

Architectural Improvements

• Generator (Multi-Receptive Field Fusion - MRF): HiFi-GAN's generator employs a module called Multi-Receptive Field Fusion (MRF). Within the generator's main structure (which still uses transposed convolutions for upsampling), the MRF module processes the intermediate features using multiple parallel residual blocks. Each residual block uses dilated convolutions with different dilation rates and kernel sizes. The outputs of these parallel blocks are then fused (typically summed). This design allows the generator to capture audio patterns across various temporal resolutions simultaneously, improving its ability to model complex waveform structures and long-range dependencies without significantly increasing computational cost.

• Discriminator (Multi-Period Discriminator - MPD + Multi-Scale Discriminator - MSD): HiFi-GAN uses a combination of two types of discriminators:

  • Multi-Period Discriminator (MPD): This is a significant innovation. The MPD consists of several sub-discriminators, each examining the input waveform at different periodic intervals. For example, one sub-discriminator might look at samples (0, p, 2p, ...), another at (0, q, 2q, ...), where p and q are different period lengths (e.g., 2, 3, 5, 7). To do this, the input waveform is reshaped by skipping samples based on the period before being fed into a 2D convolutional architecture. This structure is particularly effective at detecting subtle periodic artifacts that GANs sometimes introduce, which are often missed by standard convolutional discriminators.
  • Multi-Scale Discriminator (MSD): Similar to MelGAN's discriminator, HiFi-GAN also incorporates an MSD operating on different audio resolutions (via average pooling) to ensure realism across various time scales.

HiFi-GAN architecture overview, highlighting the Multi-Receptive Field Fusion (MRF) in the generator and the combination of Multi-Period (MPD) and Multi-Scale (MSD) discriminators.

Performance

The combination of MRF in the generator and the MPD/MSD discriminators allows HiFi-GAN to achieve state-of-the-art audio quality among GAN-based vocoders, often comparable to autoregressive models but with much superior inference speed. It effectively reduces the artifacts sometimes heard in earlier GAN vocoders, producing clean and natural-sounding speech.

Training Techniques

Training GANs for audio can be challenging:

• Stability: GAN training is notoriously sensitive to hyperparameters, initialization, and optimizer choices. Techniques like spectral normalization in the discriminator or using gradient penalties are often employed.
• Mode Collapse: The generator might learn to produce only a limited variety of sounds, failing to capture the full diversity of the training data. The multi-component discriminators in MelGAN and HiFi-GAN help mitigate this.
• Evaluation: Evaluating GANs is difficult. While objective metrics exist, perceptual evaluation through listening tests (like Mean Opinion Score - MOS) remains essential to assess the true quality and naturalness of the generated audio.

Summary

GAN-based vocoders like MelGAN and HiFi-GAN represent a significant advancement in neural waveform synthesis. By using adversarial training with carefully designed generator and discriminator architectures (incorporating multi-scale analysis, receptive field fusion, and periodic pattern detection), they achieve high-fidelity audio generation with remarkable computational efficiency. Their non-autoregressive nature makes them particularly well-suited for applications requiring low-latency speech synthesis, forming a foundation of many modern TTS pipelines. While training requires care, the resulting models offer a compelling combination of speed and quality.