Quantizing a Large Language Model aims to improve inference efficiency, reducing computational and memory requirements. However, this process introduces approximations that can affect the model's predictive quality. This chapter focuses on the necessary step of evaluating these effects.
You will learn to quantify the performance characteristics of quantized LLMs. We will cover standard metrics for evaluation, including inference latency, throughput, and memory footprint reduction (both disk size and runtime usage). You will also examine methods for assessing the impact on model accuracy, using metrics like perplexity and performance on specific downstream tasks. Techniques for benchmarking across different hardware platforms (CPUs, GPUs) and using relevant tools will be presented, allowing you to analyze the practical trade-offs between efficiency gains and potential accuracy loss.
3.1 Metrics for Quantized Model Evaluation
3.2 Measuring Inference Latency and Throughput
3.3 Assessing Memory Consumption (Disk and Runtime)
3.4 Evaluating Accuracy Degradation
3.5 Benchmarking Frameworks and Tools
3.6 Analyzing Performance on Target Hardware
3.7 Visualizing Performance Trade-offs
3.8 Hands-on Practical: Benchmarking a Quantized LLM