An Automatic Speech Recognition system operates not as a single entity but as a coordinated pipeline of distinct processing stages. Think of it like a factory assembly line, where raw material enters at one end and a finished product emerges at the other. In ASR, the raw material is a sound wave, and the finished product is text. Each station on this assembly line has a specialized job, and understanding these roles is the first step to understanding ASR.
The standard ASR pipeline consists of four main components: Feature Extraction, an Acoustic Model, a Language Model, and a Decoder. These components work in sequence to methodically transform a complex audio signal into a coherent string of words.
The standard ASR pipeline. Audio data flows from left to right, undergoing a transformation at each stage.
The first stage of the pipeline deals with the raw audio signal. A digital audio recording contains a massive amount of data, much of which is not directly useful for identifying speech. For example, it includes background noise, the speaker's emotional tone, or recording artifacts. The job of the feature extraction component is to process this raw audio and distill it into a compact, yet informative, representation.
This process converts the audio waveform into a sequence of numerical vectors, known as feature vectors. Each vector summarizes the important acoustic properties of a tiny slice of audio, typically around 25 milliseconds. These features are designed to highlight characteristics relevant to speech, like the distribution of energy across different frequencies, while discarding irrelevant information.
A very common type of feature used in ASR is the Mel-Frequency Cepstral Coefficient (MFCC). We will go into the details of how to create MFCCs and other features in Chapter 2, "Processing Audio Signals". For now, just know that this first step cleans and prepares the audio for the next stage in the pipeline.
Once we have a sequence of feature vectors, they are passed to the Acoustic Model (AM). The acoustic model is the part of the system that connects the audio to linguistic units. Its primary task is to look at a feature vector and determine which sound of the language it most likely represents.
In linguistics, the smallest units of sound that can distinguish one word from another are called phonemes. For example, the sounds /k/, /æ/, and /t/ in "cat" are phonemes. The acoustic model essentially acts as a phonetic transcriber. It takes a segment of audio features and calculates the probability of it being each possible phoneme.
The output of the AM is not a single phoneme, but a set of probabilities. For a given slice of audio, it might report:
This model is trained on large amounts of transcribed audio, learning the relationship between acoustic features and the sounds that produce them. In Chapter 3, "Acoustic Modeling", we will look at how these models are built.
The acoustic model is good at identifying individual sounds, but it has no understanding of language or context. It might confuse two acoustically similar phrases, like "recognize speech" and "wreck a nice beach". To the AM, both sequences of sounds are plausible.
This is where the Language Model (LM) comes in. The language model adds linguistic knowledge to the system. It works entirely with text and has no access to the audio. Its job is to calculate the probability of a given sequence of words. It can tell us that the phrase "recognize speech" is far more likely to occur in the English language than "wreck a nice beach".
By analyzing massive amounts of text, like books and web pages, the LM learns which word combinations are common and which are rare. It answers the question: "Given the previous words, what is the probability of the next word?" This helps the ASR system choose words that form coherent sentences. We will examine how language models work in Chapter 4, "Language Modeling".
The final component is the Decoder, which acts as the decision-maker. The decoder's job is to take the information from both the acoustic model and the language model and find the single most likely sequence of words that could have produced the original audio.
This is a complex search problem. The decoder examines a huge tree of possibilities. For each moment in time, the acoustic model provides a list of possible phonemes. These phonemes form possible words, and the words form possible sentences. The decoder's task is to find the one path through this space of possibilities that has the best combined score.
The final score for any given sentence hypothesis is a combination of two things:
The decoder uses sophisticated search algorithms, such as the Viterbi algorithm, to efficiently find the sentence that maximizes this combined score. The output of the decoder is the final text transcription, the system's best guess at what was originally said. We will discuss this process further in Chapter 5, "Decoding and Putting It All Together".
Was this section helpful?
© 2026 ApX Machine LearningEngineered with