The Complete ASR Pipeline: A Review

The complete Automatic Speech Recognition pipeline can be assembled from its components. Understanding how these parts work together is fundamental to appreciating how a machine transforms spoken sound into written text. Each stage has a distinct responsibility, and the output of one becomes the input for the next, culminating in the final transcription.

The entire process is a sophisticated chain of probability estimation and search. Fundamentally, the system is trying to answer one question: "What is the most probable sequence of words, given this audio?"

Let's walk through the entire workflow one more time, from the moment a sound wave enters the system to the moment a sentence appears.

The ASR Pipeline: From Sound to Text

The diagram below illustrates the standard architecture of a speech recognition system. We will break down each step in the sections that follow.

The flow of data through a standard Automatic Speech Recognition pipeline, from the initial audio to the final text output.

1. Audio Input and Feature Extraction

The process begins with a raw audio signal, which is a digital representation of a sound wave. As we saw in Chapter 2, this raw data is not suitable for machine learning models. It contains too much information, much of which is irrelevant for distinguishing speech sounds.

The first step, Feature Extraction, converts the audio into a compact and informative representation. The audio is segmented into short, overlapping frames (typically 25ms long). For each frame, the system calculates a set of features, with Mel-Frequency Cepstral Coefficients (MFCCs) being the most common choice. The result is a sequence of feature vectors, where each vector is a numerical summary of the frequency content of a small slice of audio.

2. The Acoustic Model: What Did It Sound Like?

The sequence of feature vectors is fed into the Acoustic Model (AM). The job of the acoustic model, which we covered in Chapter 3, is to determine the likelihood of observing these audio features given a particular linguistic unit, like a phoneme.

For each frame of audio, the AM computes the probabilities for all possible phonemes in the language. For example, for a 10ms slice of audio, it might report:

• Probability of this being the phoneme /s/ is 0.8
• Probability of this being the phoneme /t/ is 0.1
• Probability of this being the phoneme /f/ is 0.05
• ...and so on.

The output is a stream of probabilities that represents the building blocks of the sounds that were spoken.

3. The Language Model: Does It Make Sense?

In parallel, the Language Model (LM) provides an entirely different source of information. As discussed in Chapter 4, the language model knows nothing about audio. Its sole purpose is to calculate the probability of a given sequence of words. It is trained on massive amounts of text and learns which words are likely to follow others.

For instance, an N-gram language model would assign a much higher probability to the phrase "recognize speech" than to the acoustically similar but nonsensical "wreck a nice beach." It provides the linguistic context needed to resolve ambiguity.

4. The Decoder: Combining Evidence and Searching for the Best Path

This is where all the pieces come together. The Decoder is the engine that drives the search for the final transcription. It takes three inputs:

1. Acoustic Probabilities from the Acoustic Model.
2. Language Probabilities from the Language Model.
3. A Pronunciation Lexicon, which is a dictionary that maps words to their phoneme sequences (e.g., "cat" -> /k/ /ae/ /t/).

The decoder's task is to find the single sequence of words ($W$) that best explains the input audio features ($O$). It does this by finding the sequence that maximizes the product of the scores from the acoustic and language models, as expressed in the fundamental equation of speech recognition:

$$\hat{W} = \underset{W}{\mathrm{argmax}} \, P(O|W) \times P(W)$$

• $P(O|W)$ is the acoustic model score: "Given this word sequence (and its corresponding phonemes from the lexicon), what is the probability of hearing these audio features?"
• $P(W)$ is the language model score: "What is the probability of this word sequence occurring in the language?"

Because the number of possible word sequences is astronomically large, the decoder uses efficient search algorithms like the Viterbi algorithm to prune away unlikely paths and find the optimal result without evaluating every single possibility.

5. The Final Output

After the search is complete, the decoder outputs the single most likely sequence of words. This is the final transcription that the user sees. This sequence is the winner of the competition, the hypothesis that had the best combined score from both the acoustic and language models. From a sound wave to a sentence, the pipeline has successfully converted speech into text.