While intrinsic metrics like perplexity give us a measure of how well a model predicts the next token, they don't always tell the full story about a model's practical usefulness. Evaluating a model based solely on its internal language modeling capability can be insufficient for understanding its performance on specific applications.

This chapter shifts focus to extrinsic evaluation methods. We will examine how to assess Large Language Models (LLMs) by measuring their performance on concrete downstream Natural Language Processing (NLP) tasks. This approach provides a more grounded understanding of the model's capabilities in scenarios it might encounter after deployment.

You will learn about:

• The reasoning for using downstream tasks for evaluation.
• Common benchmark tasks such as text classification, question answering, and summarization.
• Standard procedures for fine-tuning pre-trained models specifically for these evaluation tasks.
• Widely used benchmark suites like GLUE and SuperGLUE.
• Evaluating models in few-shot and zero-shot settings, assessing their ability to generalize with minimal task-specific examples.
• Considerations for designing custom evaluation tasks tailored to unique requirements.

By the end of this chapter, you will understand how to measure an LLM's effectiveness through its application to specific, practical NLP problems, complementing the insights gained from intrinsic metrics.