While a standard spectrogram gives us a detailed view of a signal's frequency content over time, it has a significant limitation: its frequency axis is linear. This means the distance between 100 Hz and 200 Hz is treated the same as the distance between 8000 Hz and 8100 Hz. Human hearing, however, does not work this way. Our perception of pitch is more logarithmic; we are far more sensitive to changes in lower frequencies than in higher ones.
To align our audio features with this psychoacoustic property, we introduce filter banks spaced on the Mel scale. This approach leads to log-mel spectrograms, a feature representation that has become a foundation of modern, high-performance ASR systems.
The Mel scale is a perceptual scale of pitches judged by listeners to be equal in distance from one another. The name "Mel" comes from the word melody to indicate that the scale is based on pitch comparisons. The relationship between frequency in Hertz () and the Mel scale () is given by the following formula:
As the formula shows, the mapping is nearly linear for frequencies below 1000 Hz but becomes increasingly logarithmic for higher frequencies. This effectively mirrors how our ears work, grouping high frequencies together while preserving resolution in the lower, more perceptually important frequency ranges for speech.
The Mel scale compresses high frequencies, better reflecting the non-linear nature of human pitch perception compared to the linear Hertz scale.
A Mel filter bank is a set of triangular filters that we apply to the power spectrogram. These filters have two main characteristics:
A visualization of triangular filters. Notice how the filters become wider and more spread out as the frequency increases, mimicking the Mel scale.
To create a mel spectrogram, we multiply the values in each frame of our power spectrogram by the weights of each triangular filter and sum the results. If a power spectrogram frame has 513 frequency bins, and we use a filter bank with 80 filters, this process will transform the 513-dimensional vector into an 80-dimensional one for that time step. This operation effectively groups the energy from the linear frequency bins into a smaller number of perceptually meaningful Mel frequency bins.
The final step is to take the logarithm of the energies from the Mel filter bank. This gives us the log-mel spectrogram. We do this for a simple but important reason: human perception of loudness is also logarithmic, not linear. A sound that is objectively twice as powerful is not perceived as being twice as loud. Taking the log compresses the dynamic range of the filter bank energies, making the resulting features more aligned with our hearing.
The complete process is summarized below.
The pipeline for generating log-mel spectrogram features from a raw audio signal.
In the next section, we will discuss Mel Frequency Cepstral Coefficients (MFCCs), which for a long time were the standard input feature for ASR. MFCCs are created by applying a final transformation (the Discrete Cosine Transform, or DCT) to the log-mel spectrogram. This DCT step decorrelates the features, which was very helpful for the statistical models used in older ASR systems, like Gaussian Mixture Models.
However, for modern deep learning models, log-mel spectrograms are often the preferred input feature for several reasons:
Information Richness: The DCT step in MFCC creation is a lossy compression. It discards some information from the spectrogram that deep neural networks, particularly CNNs and Transformers, might find useful for distinguishing phonemes. By stopping at the log-mel spectrogram, we provide the model with a richer, more detailed representation of the audio.
Structural Advantage for CNNs: Log-mel spectrograms are essentially 2D images, where one axis is time and the other is Mel frequency. This structure is perfectly suited for Convolutional Neural Networks (CNNs), which are designed to learn patterns from grid-like data. A CNN can apply 2D filters to the spectrogram to detect shapes and textures corresponding to phonetic events, just as it would detect edges or shapes in a photograph.
End-to-End Learning: Modern architectures are powerful enough to learn the optimal feature representations themselves. Providing them with less-processed features like log-mel spectrograms allows the model to learn the most relevant correlations and transformations directly from the data, rather than relying on the fixed transformation of the DCT.
In practice, generating these features is straightforward using libraries like Librosa. For instance, the function librosa.feature.melspectrogram takes a raw waveform and efficiently computes the log-mel spectrogram, abstracting away the underlying STFT, windowing, and filter bank application. We will put this into practice in the hands-on exercise at the end of this chapter.
Was this section helpful?
librosa.feature.melspectrogram documentation, Brian McFee and the Librosa developers, 2024 - Official documentation for the Python library function used to compute Mel spectrograms, essential for practical implementation.© 2026 ApX Machine LearningEngineered with