To an ASR system, a spoken sentence is initially just a complex waveform. To transcribe it into text, the system must recognize the underlying linguistic patterns within that signal. Phonetics and phonology provide a foundational framework for understanding the linguistic structures of speech. Instead of treating audio as an unstructured stream of data, analysis can focus on the fundamental components of human speech.
The most basic, meaning-distinguishing unit of sound in a language is called a phoneme. Think of phonemes as the atomic elements of spoken language. Changing a phoneme in a word changes the word's meaning entirely.
For example, consider the words "pat," "bat," and "cat." The only difference between them is the initial sound. The sounds represented by /p/, /b/, and /k/ are distinct phonemes in English because substituting one for another creates a new word. The slashes, as in /p/, are used to denote a phoneme as an abstract sound unit, distinct from the letters of the alphabet that might represent it.
A language has a finite set of phonemes. American English, for example, has around 44 phonemes, which include consonants, short vowels, long vowels, and diphthongs (vowels that glide from one position to another, like the /aɪ/ sound in "buy"). An ASR system's first major task, whether explicitly or implicitly, is to identify sequences of these phonemes from an audio signal.
While phonemes are the abstract units, the sounds we actually produce are called phones. A single phoneme can be physically pronounced in slightly different ways depending on its context within a word. These predictable variations of a single phoneme are called allophones.
A classic example in English is the phoneme /p/.
/p/ sound is aspirated. This means it is followed by a small puff of air. The phonetic notation for this allophone is [pʰ]./p/ sound is unaspirated. There is no accompanying puff of air. The notation for this allophone is [p].To a native English speaker, [pʰ] and [p] sound like the same "p" sound. Our brains automatically group them into the single phoneme /p/. However, to a machine analyzing a waveform, these two sounds are acoustically different. The presence or absence of that puff of air creates a measurable difference in the audio signal.
This relationship illustrates a core challenge in speech recognition. The ASR model must learn that these acoustically distinct allophones, [pʰ] and [p], both map to the same phoneme /p/ and, ultimately, to the same letter 'p' in the final transcription.
The phoneme
/p/is an abstract sound category. Depending on its position in a word, it can be realized as different allophones, such as the aspirated[pʰ]in "pin" or the unaspirated[p]in "spin".
The variation in speech sounds is further complicated by coarticulation. This is the phenomenon where the pronunciation of a sound is influenced by its neighboring sounds. Human speech is not a sequence of discrete, perfectly separated phones. Instead, the sounds blend together.
For example, say the words "ten" and "tenth" out loud and pay attention to the position of your tongue when you make the /n/ sound.
/θ/) that follows.This anticipatory movement changes the acoustic properties of the /n/ sound. Coarticulation means that the same phoneme can have a different acoustic signature almost every time it is uttered.
Understanding phonemes, allophones, and coarticulation is not just an academic exercise. It highlights the primary difficulty in speech recognition: the immense variability in the speech signal. An ASR system cannot simply memorize a single acoustic pattern for each sound. It must learn a flexible representation that can account for:
Modern deep learning models for ASR are powerful because they can learn to handle this variability directly from data. By training on thousands of hours of speech from many different speakers, these models learn to map diverse acoustic patterns to the correct linguistic units, whether they are phonemes or, more commonly in end-to-end systems, characters or words. The feature extraction techniques we will discuss in the next chapter are designed to create a representation of speech that is more resilient to this type of variation.
Was this section helpful?
© 2026 ApX Machine LearningEngineered with