Having established the fundamental challenges in deploying large language models efficiently, this chapter concentrates on quantization, a primary method for reducing model size and accelerating inference. While basic quantization offers benefits, achieving significant gains without sacrificing model accuracy requires more sophisticated approaches.

This chapter moves past introductory concepts to cover advanced quantization methodologies specifically applied to LLMs. You will learn to:

• Implement and analyze advanced Post-Training Quantization (PTQ) calibration methods and Quantization-Aware Training (QAT) tailored for large transformer models.
• Investigate extreme quantization techniques, pushing precision below 4 bits (e.g., NF4, FP4) and exploring binary/ternary representations.
• Design mixed-precision strategies, applying different levels of quantization ($int8$, $fp4$, $int4$, etc.) to different model components.
• Understand how specific hardware features accelerate quantized computations.
• Establish rigorous protocols for evaluating the trade-offs between compression, inference speed, and model fidelity.

We will examine the practical application of these techniques, including outlier handling, accuracy preservation, and the interaction between quantization and hardware accelerators. The chapter concludes with a hands-on exercise implementing PTQ and QAT.