After adapting a Large Language Model through fine-tuning, assessing its actual performance is a critical step. Simply applying standard NLP metrics often fails to capture the subtleties of generative model behavior, especially for specialized tasks or domains. This chapter provides the methods needed for a comprehensive evaluation.

We will address the shortcomings of conventional metrics and introduce techniques specifically suited for evaluating fine-tuned LLMs. You will learn systematic approaches to assess instruction adherence, check for factual accuracy and the generation of unsupported claims (hallucinations), and analyze potential biases within model outputs. Furthermore, we cover techniques for testing model robustness against varied inputs, the significance of model calibration, and the essential roles of qualitative analysis and structured human feedback in understanding true model capabilities and limitations.