The Concept of Perplexity

Language models assign probabilities to sequences of words. A fundamental question in their application is how to measure their effectiveness. For instance, when comparing different models, such as a bigram model and a trigram model, a quantitative method is needed to determine which one is better at predicting text. This evaluation is performed using a metric called perplexity.

Measuring a Model's "Surprise"

At its core, perplexity is a measure of how well a probability model predicts a sample. For a language model, it measures how uncertain the model is when predicting the next word in a sequence. You can think of it as a measure of "surprise." A language model that is good at its job will be less "surprised" by a typical sentence from a test set. This lack of surprise translates to assigning a higher probability to that sentence.

A lower perplexity score indicates that the language model is better at predicting the text. A higher perplexity score indicates that the model is more "perplexed" by the text and, therefore, is not a good fit for it.

What a Perplexity Score Represents

The most intuitive way to interpret a perplexity score is to think of it as the effective number of choices the model has for the next word.

For example, if a language model has a perplexity of 20 on a given test set, it means that on average, the model is as confused about predicting the next word as if it had to choose uniformly from 20 different words. A better model would have a lower perplexity, say 10, which would mean its uncertainty is equivalent to choosing between only 10 words.

A perfect model that always knows the next word would have a perplexity of 1 (it has only one choice). Of course, this is not possible in practice with natural language.

The following diagram illustrates this idea. A model with low perplexity has a smaller effective branching factor, meaning it has narrowed down the likely options more effectively than a model with high perplexity.

The low perplexity model on the left is more certain about the next word, while the high perplexity model on the right has a much larger set of roughly equally probable choices.

The Math Behind Perplexity

Perplexity is derived directly from the probability assigned to a test set by the language model. If a test set $W$ consists of a sequence of words $w_1, w_2, \dots, w_N$, the perplexity is calculated as the inverse of the geometric mean of the probabilities of the words in the sequence.

The formula is:

$$\text{Perplexity}(W) = P(w_1, w_2, \dots, w_N)^{-\frac{1}{N}}$$

Let's break this down:

1. $P(w_1, w_2, \dots, w_N)$: This is the probability of the entire test sentence, calculated by the language model. For an N-gram model, this is found by chaining conditional probabilities, such as $P(w_1) \times P(w_2 | w_1) \times P(w_3 | w_2) \dots$. This value is typically very, very small.
2. The Exponent $(-\frac{1}{N})$: This part does two important things.
   • The $\frac{1}{N}$ normalizes the probability by the number of words ($N$) in the test set. This allows us to compare perplexity scores on test sets of different lengths.
   • The negative sign effectively inverts the probability. Because probabilities are between 0 and 1, a higher probability will result in a lower perplexity score, which is exactly what we want. A better model assigns higher probability and gets a lower (better) score.

In practice, because multiplying many small probabilities can lead to numerical underflow (the computer rounds the number to zero), calculations are often done using log probabilities. The formula using logs is equivalent but more stable for computers to handle.

An Example Scenario

Imagine you have two language models you want to compare for an ASR system designed to transcribe financial news.

• Model A: Trained on a large corpus of financial news articles from the Wall Street Journal.
• Model B: Trained on a large corpus of children's stories.

Now, you evaluate both models on a test set of unseen financial news headlines. Let's take the sentence, "The Federal Reserve raised interest rates."

• Model A would likely assign a high probability to this sequence. Words like "Federal," "Reserve," "interest," and "rates" are common and appear together often in its training data. Its perplexity score on this test set would be relatively low.
• Model B would be very "surprised" by this sentence. The vocabulary and word sequences are completely unlike those found in children's stories. It would assign an extremely low probability to this sentence, resulting in a very high perplexity score.

By comparing the perplexity scores, you can clearly conclude that Model A is the superior choice for your ASR application. This metric provides a formal way to confirm our intuition, making it an essential tool for developing and selecting language models.