Hidden Markov Models (HMMs) for Sequential Data

Speech is inherently sequential. The sounds in the word "cat" appear in a specific order, and this order is what gives the word its meaning. A model that only identifies the properties of individual sounds, such as Gaussian Mixture Models, misses this fundamental structure. It is like having a bag of letters. you can see which letters are there, but you cannot form words or sentences. To correctly interpret speech, we need a system that understands sequences. This is the role of Hidden Markov Models (HMMs).

An HMM is a statistical tool designed for modeling sequential data where the underlying process generating the data is not directly visible. Imagine you are in a windowless room and want to guess the weather outside. The weather itself, whether it is "Sunny" or "Rainy," is a hidden state. You cannot see it. However, you can observe whether your colleague who comes in from outside is carrying a wet umbrella. This is your observation. By tracking these observations over time, you can infer the most likely sequence of weather conditions that occurred.

Applying HMMs to Speech

This model translates directly to speech recognition:

• The Hidden States: The phonemes of a language are the hidden states. We never directly "see" a perfect /k/ or /æ/ in a raw audio waveform.
• The Observations: The feature vectors, such as MFCCs, that we extract from short frames of audio are our observations. They are the acoustic evidence we have.

The HMM provides a mathematical framework to find the most probable sequence of hidden phonemes given the sequence of observed audio features.

Components of a Hidden Markov Model

To build this framework, an HMM relies on three main components:

1. A Set of States ($S$): For speech, we don't just use one state per phoneme. A single phoneme's sound changes from its beginning to its end. Therefore, we typically model each phoneme with a small chain of states, usually three: a beginning, a middle, and an end state. This allows the model to better capture the dynamic nature of speech sounds.

2. Transition Probabilities ($A$): These define the probability of moving from one state to another. For example, there is a high probability of moving from the "beginning" state of /k/ to its "middle" state. There is also a probability of moving from the final state of /k/ to the first state of the next phoneme, like /æ/. These probabilities enforce the legal ordering of sounds. States can also transition back to themselves, which allows the model to account for a phoneme being spoken slower or faster.

3. Emission Probabilities ($B$): This is the probability of producing a particular observation (an audio feature vector) while in a specific state. It answers the question: if we are in the state representing the middle of the phoneme /æ/, what is the probability $P(\mathrm{audio\_features} | \mathrm{state æ-mid})$ of observing the specific MFCC vector we just calculated?

The diagram below illustrates a simple HMM for the word "cat," which consists of the phonemes /k/, /æ/, and /t/. Each phoneme is modeled with three states, and the arrows represent possible transitions.

An HMM representing the word "cat" (/k/ /æ/ /t/). The model transitions through sub-states for each phoneme. Self-loops allow the model to spend variable amounts of time in each state, accounting for differences in speaking speed.

By combining transition and emission probabilities, an HMM can calculate an overall score for any given sequence of phonemes against the input audio. The goal of the ASR system becomes finding the single path through all possible phoneme states that has the highest probability of generating the observed audio features. This is a complex search problem, but it can be solved efficiently with a special procedure known as the Viterbi algorithm, a topic we will visit in a later chapter.

The HMM provides the sequential "scaffolding" for the acoustic model. It masterfully handles the "when" of speech sounds, modeling their order and duration. However, the HMM framework itself doesn't define how to calculate the emission probabilities, the $P(\mathrm{audio\_features} | \mathrm{state})$ part. It needs a way to evaluate how well a given audio frame matches a phoneme state. As you will see in the next section, this is precisely where GMMs come back into the picture, forming a powerful partnership with HMMs.