After investing significant compute and data resources into training a large language model, the next step is to measure how well it actually performs its fundamental task: modeling language. Evaluating these complex systems requires specific techniques. One category of evaluation focuses directly on the model's ability to predict sequences of text, without necessarily testing it on downstream applications.

This chapter concentrates on these intrinsic evaluation methods. We will look at metrics derived directly from the model's probabilities assigned to text sequences. The most common intrinsic metric is perplexity, which quantifies how well a probability model predicts a sample. It is closely related to the cross-entropy loss used during training. A lower perplexity score generally indicates that the model is better at predicting the test data, meaning it assigns higher probabilities to the observed sequences. For a sequence $W = w_1, w_2, ..., w_N$, perplexity can be expressed based on the model's assigned probabilities $p(w_i | w_{<i}; \theta)$ as:

$PPL(W) = \exp\left(-\frac{1}{N}\sum_{i=1}^N \log p(w_i | w_{<i}; \theta)\right)$

Understanding perplexity provides a baseline assessment of the model's language modeling quality.

In this chapter, you will learn:

• The definition of perplexity and its calculation from model outputs.
• How to interpret perplexity values and understand their relative meaning and limitations.
• Alternative related metrics like bits per character or bits per word.
• The effect different tokenization strategies can have on calculated perplexity scores.