While Automatic Speech Recognition (ASR) focuses on transcribing spoken audio into text, Text-to-Speech (TTS) synthesis undertakes the complementary task: generating audible speech from input text. Although modern end-to-end neural systems often learn mappings directly, understanding the traditional logical components provides a solid foundation for analyzing and designing sophisticated TTS pipelines. These components work sequentially to transform text into a natural-sounding waveform.

Let's dissect the typical stages involved in a parametric or concatenative TTS system, keeping in mind that neural approaches may integrate or re-imagine these steps.

Text Processing (Frontend)

The journey from text to speech begins with understanding and standardizing the input text. This frontend processing ensures the text is unambiguous and structured appropriately for subsequent stages.

1. Text Normalization

Raw text often contains non-standard words like numbers, abbreviations, symbols, and punctuation that need conversion into their fully spelled-out, speakable forms. This process, known as text normalization (TN) or non-standard word (NSW) processing, is crucial for intelligibility.

Examples:

$12.50 -> "twelve dollars and fifty cents"
Dr. Smith -> "Doctor Smith"
St. Louis -> "Saint Louis"
10 Downing St. -> "Ten Downing Street"
1998 -> "nineteen ninety-eight"

Text normalization can be complex, involving regular expressions, finite-state transducers (FSTs), or increasingly, machine learning models. It's highly language-dependent and requires careful handling of ambiguity (e.g., "St." can mean "Saint" or "Street"). Errors in TN propagate directly into the synthesized speech.

2. Linguistic Analysis

Once normalized, the text undergoes linguistic analysis to extract features relevant for pronunciation and prosody (intonation, rhythm, stress).

Grapheme-to-Phoneme (G2P) Conversion: This converts orthographic text (graphemes) into phonetic representations (phonemes). For instance, the word "synthesis" might be converted to a phoneme sequence like /s ɪ n θ ə s ɪ s/ (using ARPABET notation). This can be achieved using:
- Pronunciation Dictionaries (Lexicons): Large lookup tables mapping words to their phonetic transcriptions.
- Rule-Based Systems: Hand-crafted rules for pronunciation patterns.
- Machine Learning Models: Statistical or neural models trained on dictionary data to predict pronunciations for known and unknown (out-of-vocabulary, OOV) words.
Part-of-Speech (POS) Tagging: Identifying the grammatical role of each word (noun, verb, adjective, etc.) helps resolve pronunciation ambiguities (e.g., "reCORD" (verb) vs. "REcord" (noun)) and informs prosody prediction.
Prosody Prediction: While basic systems might use heuristics, advanced systems explicitly predict prosodic features. This can include:
- Phrase Break Prediction: Determining where pauses should occur.
- Accent/Stress Prediction: Identifying words or syllables that should be emphasized.
- Intonation Contour Generation: Predicting the fundamental frequency ( $F_0$ ) contour, which largely determines the perceived melody of speech. This often involves analyzing sentence structure (e.g., questions vs. statements).

The output of the frontend is typically a sequence of phonemes enriched with linguistic and prosodic information.

Acoustic Feature Generation (Backend/Synthesizer)

The backend takes the processed linguistic features from the frontend and generates intermediate acoustic representations.

3. Duration Modeling

Each phoneme (or other linguistic unit) doesn't have a fixed duration. The length varies significantly based on phonetic context, position within a word or phrase, speaking rate, and emphasis. The duration model predicts the duration (often in milliseconds or frames) for each unit in the input sequence. Accurate duration modeling is critical for achieving natural speech rhythm. Traditional systems often used HMMs or decision trees, while modern neural systems typically incorporate duration predictors within their architectures (e.g., as seen in FastSpeech or integrated into attention mechanisms).

4. Acoustic Feature Prediction

This is the core synthesis step where the system maps the linguistic features (phonemes, durations, prosodic targets) to a sequence of acoustic features. These features aim to capture the spectral envelope and excitation characteristics of the target speech but in a compressed, intermediate format.

Feature Types: A common choice is mel-spectrograms, which represent the power spectrum of the audio mapped onto the non-linear mel scale of human frequency perception. Other features like cepstral coefficients or linguistic-acoustic features derived from HMM states have also been used.
Modeling Approaches:
- Classical Parametric: HMM-based systems generate parameters (like spectral features and $F_0$ ) based on statistical models trained on context-dependent states.
- Concatenative: Selects and concatenates pre-recorded speech units (diphones, phonemes) from a large database. While capable of high quality for domain-specific voices, it suffers from audible discontinuities and lacks flexibility.
- Neural Sequence-to-Sequence: Models like Tacotron or Transformer TTS learn a complex, non-linear mapping directly from input linguistic features (often characters or phonemes) to acoustic features (typically mel-spectrograms) using encoder-decoder architectures with attention. These models implicitly handle duration and context modeling. Non-autoregressive models like FastSpeech use explicit duration predictors alongside feed-forward transformers for faster generation.

The output of this stage is a frame-by-frame sequence of acoustic feature vectors.

Waveform Generation (Vocoder)

The acoustic features generated by the synthesizer are not yet an audio waveform. The final step uses a vocoder (voice coder/decoder) to convert these features into audible sound pressure waves.

5. Vocoding

Traditional Vocoders: Signal processing techniques like Source-Filter models (e.g., STRAIGHT, WORLD) or overlap-add methods based on the Short-Time Fourier Transform (STFT) magnitude (e.g., Griffin-Lim algorithm) were common. These often produce intelligible but somewhat artificial or "buzzy" speech, especially at lower feature resolutions.
Neural Vocoders: Modern TTS relies heavily on neural vocoders, which are deep generative models trained to synthesize high-fidelity waveforms conditioned on acoustic features. Examples include:
- Autoregressive Models: WaveNet, WaveRNN generate audio sample by sample, achieving high quality but often slow inference.
- Flow-Based Models: WaveGlow, FloWaveNet use normalizing flows for parallel generation.
- GAN-Based Models: MelGAN, HiFi-GAN, Parallel WaveGAN use Generative Adversarial Networks for efficient and high-quality waveform synthesis.
- Diffusion Models: Recent approaches apply diffusion probabilistic models to waveform generation.

These advanced vocoders (discussed further in Chapter 5) are a significant reason for the dramatic improvements in TTS naturalness over the past decade.

System Overview

The interaction between these components can be visualized as a pipeline:

A typical pipeline for Text-to-Speech synthesis, showing the progression from input text through frontend processing, backend synthesis (duration and acoustic feature prediction), and final waveform generation via a vocoder. Modern end-to-end systems might integrate several of these stages.

While presented sequentially, these components interact closely. Errors or limitations in early stages (like incorrect phonemization or unnatural duration predictions) significantly impact the final output quality. As we explore advanced TTS models in Chapter 4, we will see how end-to-end neural networks aim to learn the entire mapping from text to acoustic features, or sometimes even directly to waveforms, potentially simplifying the pipeline but often requiring larger datasets and careful training strategies. Understanding these fundamental building blocks remains essential for diagnosing issues, customizing systems, and appreciating the complexity involved in generating human-like speech.