Feature Normalization Techniques

Speech recognition systems rely on extracted features, such as MFCCs or log-mel spectrograms. These raw feature values can vary significantly due to factors unrelated to the spoken content, such as microphone type, recording distance, and background noise level. Feature normalization is a preprocessing step designed to address this variability, ensuring that the model learns the phonetic content of speech rather than incidental recording conditions. By scaling features to a consistent range, normalization helps stabilize the training process and improve the model's ability to generalize to new, unseen data.

Cepstral Mean and Variance Normalization (CMVN)

The most common normalization technique used in speech recognition is Cepstral Mean and Variance Normalization, or CMVN. The objective of CMVN is to transform the feature vectors so that they have a mean of zero and a variance of one. This process, also known as standardization, removes shifts in the mean and variance of the features that can be caused by the recording channel or speaker characteristics.

Applying CMVN involves calculating the mean and standard deviation for each feature coefficient across a set of frames and then using these statistics to normalize each frame. This can be done in a few different ways, depending on the scope of the statistics calculation.

• Utterance-level CMVN: The mean and variance are calculated and applied on a per-utterance basis. This is effective at removing variations specific to a single recording, such as a sudden change in volume. It is simple to apply because each utterance is self-contained.
• Speaker-level CMVN: Statistics are computed over all utterances from a single speaker. This is useful for adapting to the unique vocal characteristics of an individual but requires speaker labels for your data, which may not always be available.
• Global CMVN: The mean and variance are calculated over the entire training dataset. These fixed statistics are then saved and applied to all data, including the validation and test sets. This is a very common approach in ASR pipelines as it ensures consistent scaling across all parts of the project.

The diagram below illustrates the process for utterance-level CMVN, where each utterance's feature matrix is normalized independently.

The process of utterance-level Cepstral Mean and Variance Normalization.

The Normalization Formula

For a given feature coefficient dimension $d$ (for example, the 4th MFCC coefficient), the mean $\mu_d$ is calculated across all $T$ time frames of an utterance or set of utterances:

$$\mu_d = \frac{1}{T} \sum_{t=1}^{T} c_t(d)$$

Here, $c_t(d)$ is the value of the $d$-th coefficient at time frame $t$. The standard deviation $\sigma_d$ is calculated as the square root of the variance:

$$\sigma_d = \sqrt{\frac{1}{T} \sum_{t=1}^{T} (c_t(d) - \mu_d)^2}$$

Finally, each coefficient $c_t(d)$ is normalized to produce the new feature value $\hat{c}_t(d)$:

$$\hat{c}_t(d) = \frac{c_t(d) - \mu_d}{\sigma_d}$$

This calculation is performed independently for each coefficient in the feature vector. For example, if you are using 13 MFCCs, you will compute 13 separate means and 13 separate standard deviations.

The chart below shows a histogram of values for a single feature coefficient before and after CMVN. Before normalization, the feature distribution is centered around a mean of 5.0. After applying CMVN, the distribution is centered at 0 with a standard deviation of 1.

Effect of CMVN on the distribution of a single feature coefficient. The normalized feature is centered at zero.

Practical Notes

When implementing a feature extraction pipeline, the choice of normalization strategy has practical implications.

If you choose global CMVN, it is extremely important to compute the mean and standard deviation statistics only from the training data. Saving these statistics and applying them to the validation and test sets prevents information from the test set from "leaking" into your training process, which would give you an overly optimistic evaluation of your model's performance.

For utterance-level CMVN, the process is simpler as each file is handled independently. This can be beneficial for applications where the recording conditions are expected to change frequently.

While modern neural network architectures often include internal normalization layers like BatchNorm or LayerNorm, applying CMVN directly to the input features remains a common and beneficial practice. It provides a clean, standardized input that can reduce the burden on the initial layers of the network, often leading to faster convergence and better overall performance. This step ensures that the acoustic model can focus on learning the mapping from phonetically relevant patterns to text, without being distracted by extrinsic acoustic variability.