Evaluation of Synthesized Audio Quality

Evaluating the output of neural vocoders is essential for understanding their performance and comparing different models. Unlike traditional vocoders where limitations like buzziness or muffling were often obvious, neural vocoders aim for perceptual indistinguishability from real human speech. This higher bar necessitates more sophisticated evaluation methods, encompassing both automated signal analysis and human perceptual judgments. Simply generating a waveform isn't enough; we need to quantify how good that waveform sounds.

Objective Evaluation Metrics

Objective metrics analyze the synthesized waveform mathematically, comparing it against a ground truth (original) recording. They offer repeatable, automated assessment but may not always align perfectly with human perception.

• Log-Spectral Distance (LSD): This metric measures the average difference between the log power spectra of the ground truth and synthesized audio signals, typically computed frame by frame. It quantifies how similar the spectral content is. Lower LSD values indicate greater similarity. The formula is:

  $$LSD = \sqrt{\frac{1}{T} \sum_{t=1}^{T} \frac{1}{K} \sum_{k=1}^{K} (10 \log_{10} |S_t(k)|^2 - 10 \log_{10} |\hat{S}_t(k)|^2)^2}$$

  Here, $S_t(k)$ and $\hat{S}_t(k)$ represent the spectral magnitudes of the original and synthesized signals, respectively, at time frame $t$ and frequency bin $k$. $T$ is the total number of frames, and $K$ is the number of frequency bins.

• Mel-Cepstral Distortion (MCD): This is a popular metric in speech synthesis that measures the Euclidean distance between the mel-cepstral coefficients (MCCs) of the ground truth and synthesized audio. Since MCCs are derived based on the human auditory system's frequency perception (using the Mel scale), MCD is considered more perceptually relevant than raw spectral distance. It's usually expressed in decibels (dB), with lower values being better. The calculation often involves Dynamic Time Warping (DTW) to align the MCC sequences before computing the distance:

  $$MCD [dB] = \frac{10}{\ln 10} \sqrt{2 \sum_{d=1}^{D} (mcc_d - \hat{mcc}_d)^2}$$

  This formula calculates the distortion for a specific dimension $d$, up to dimension $D$ (often 13 to 40), after alignment. The final MCD is averaged over all frames.

• Signal-to-Noise Ratio (SNR) / Segmental SNR (SegSNR): SNR measures the ratio of the power of the original signal to the power of the error (difference between original and synthesized). Higher SNR is generally better. Segmental SNR calculates SNR over short segments (e.g., 20-30 ms) and averages them, which often correlates better with perceived quality than a global SNR, as it prevents quiet segments from being dominated by louder ones in the overall calculation. However, SNR metrics can be overly sensitive to phase differences and may not fully capture perceptual naturalness.

• PESQ (Perceptual Evaluation of Speech Quality): Defined in ITU-T recommendation P.862, PESQ is an algorithm designed to predict subjective listening quality. It compares the original reference signal to the degraded (synthesized) signal through a complex auditory transform model. The output score typically ranges from -0.5 to 4.5, approximating a Mean Opinion Score (MOS), with higher scores indicating better perceptual quality. PESQ is widely used but requires the ground truth reference signal.

• STOI (Short-Time Objective Intelligibility): This metric aims to predict speech intelligibility by measuring the correlation between the temporal envelopes of the clean reference speech and the processed speech within short time frames across different frequency bands. It produces a score between 0 and 1, where higher values indicate better intelligibility. While primarily designed for noise suppression evaluation, it can provide insights into the clarity of synthesized speech.

It's important to remember that objective metrics provide valuable quantitative data but don't tell the whole story. A model might achieve excellent scores on LSD or MCD but still produce subtle artifacts that a human listener finds unpleasant.

Subjective Evaluation Metrics

Subjective tests involve human listeners rating the quality of synthesized audio. They are considered the definitive measure of perceptual quality but are more time-consuming and expensive to conduct properly.

• Mean Opinion Score (MOS): This is the most common subjective test. A group of listeners rates audio samples on an absolute quality or naturalness scale, typically from 1 to 5.

  • 5: Excellent (Indistinguishable from natural speech)
  • 4: Good (Some minor imperfections)
  • 3: Fair (Noticeable imperfections, but not annoying)
  • 2: Poor (Annoying imperfections)
  • 1: Bad (Completely unnatural or unintelligible)

  MOS tests require careful setup: a controlled listening environment (e.g., quiet room, headphones), a sufficiently large and diverse group of listeners, clear instructions, and statistical analysis of the results (including confidence intervals) to ensure reliability.

• Comparison Tests (A/B or A/B/X): Instead of absolute ratings, listeners compare two (A/B) or more samples directly. In an A/B test, listeners state their preference between sample A and sample B, or indicate no preference. In an A/B/X test, listeners hear A, B, and then X (which is either A or B), and must identify whether X matches A or B. This helps gauge preference and discriminability between systems. CMOS (Comparison MOS) scores are often derived from A/B tests, indicating the average preference strength on a scale (e.g., -3 to +3).

• MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor): Defined by ITU-R recommendation BS.1534, MUSHRA is useful when comparing several systems that might be close in quality. Listeners are presented with multiple stimuli simultaneously: the original reference (hidden), the outputs from various systems being tested, and one or more low-quality "anchors". Listeners rate each stimulus (except the explicit reference, if provided) on a continuous scale from 0 to 100 relative to the reference. This setup helps listeners calibrate their ratings and provides a sensitive measure for high-quality audio where differences might be subtle.

Subjective tests provide the most direct assessment of how humans perceive the synthesized speech, capturing aspects like naturalness, pleasantness, and the presence of artifacts that objective metrics might miss.

Example Mean Opinion Scores (MOS) comparing a traditional vocoder (Griffin-Lim) with several neural vocoders. Higher scores indicate better perceived naturalness. Note that actual scores depend heavily on the specific model, training data, and test conditions.

Practical Considerations

When evaluating vocoders, consider these points:

1. Reference Quality: The quality of the ground truth audio is fundamental. Objective metrics require a clean reference signal, and subjective tests rely on it for comparison (either explicit or implicit).
2. Vocoder vs. End-to-End: Are you evaluating the vocoder in isolation (using ground truth acoustic features) or as part of a full TTS system (using predicted acoustic features)? The latter reflects real-world performance but mixes vocoder quality with the quality of the acoustic model. Evaluating both ways is often informative.
3. Data Diversity: Ensure evaluation covers a range of speakers, speaking styles, and content to assess the vocoder's generalizability.
4. Trade-offs: High-quality vocoders (like autoregressive models) can be computationally expensive. Evaluation might also need to consider inference speed, model size, and suitability for real-time deployment alongside perceptual quality.

Ultimately, a comprehensive evaluation strategy combines automated objective metrics for rapid iteration and diagnostics with rigorous subjective testing to confirm genuine improvements in perceptual quality and naturalness. Choosing the appropriate mix depends on the project's goals and available resources.