Mapping Sounds to Phonemes

At its core, the acoustic model acts as a bridge. Audio waveforms are transformed into sequences of feature vectors, such as MFCCs. Each vector provides a numerical snapshot of the sound's characteristics over a very short period, typically 10 to 25 milliseconds. The primary objective is to translate these numbers into the fundamental sounds of language, referred to as phonemes.

The central task of the acoustic model is to determine the most likely phoneme for each of these feature vectors. However, this isn't a simple one-to-one lookup. The way a person says /t/ in the word "top" can be acoustically different from the /t/ in "water" or "stop". Accents, speaking speed, and even the speaker's mood can alter the sound. Therefore, the acoustic model must think in terms of probabilities.

Instead of making a definitive decision, the model calculates a likelihood score for every possible phoneme in a language. For a single frame of audio features, it asks:

• What is the probability that these features correspond to the phoneme /k/?
• What is the probability they correspond to /æ/?
• What is the probability they correspond to /t/?
• And so on, for all phonemes.

This process generates a probability distribution for each time step. The result is not a single phoneme, but a list of possibilities, each with a score. This relationship is often expressed mathematically as the likelihood $P(\text{features} | \text{phoneme})$. This formula reads as "the probability of observing a specific set of audio features, given that a particular phoneme was spoken." The model computes this for every phoneme, allowing it to score how well each one "fits" the observed audio data.

The following diagram illustrates this mapping for a single time frame. A vector of MFCC features is fed into the acoustic model, which in turn outputs a set of probabilities for each candidate phoneme.

The acoustic model computes the probability of each phoneme given a set of audio features for a specific time frame.

This probabilistic output is significant. If two phonemes sound very similar, like /p/ and /b/, the model might assign high probabilities to both. For instance, for a given sound, it might report $P(\text{features} | \text{/p/}) = 0.45$ and $P(\text{features} | \text{/b/}) = 0.40$. The acoustic model itself doesn't have to make the final choice. It simply provides these scores.

By processing the entire sequence of feature vectors from an audio clip, the acoustic model produces a corresponding sequence of these probability distributions. This rich, time-aligned phonetic information becomes the primary input for the next stages of the ASR pipeline. The system will later use a language model and a decoder to weigh these possibilities, look at the surrounding context, and ultimately decide if the speaker said "big" or "pig".

In the sections that follow, we will look at how this model is built, starting with traditional statistical methods and moving toward modern neural network approaches.