Executing machine learning models often demands significant computational resources and memory bandwidth. Reducing the numerical precision of model weights and activations offers a path to lower latency, smaller memory footprint, and reduced power consumption. This performance gain comes at the cost of potential accuracy degradation, requiring careful management through specialized techniques.

This chapter focuses on the compiler and runtime strategies needed to implement and optimize models using low-precision arithmetic, primarily 8-bit integers ($INT8$) and emerging lower-precision floating-point formats like $FP8$. We will examine how quantization principles, including mapping schemes and calibration, are represented within compiler intermediate representations (IRs). You will learn about compiler flows supporting both Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ), including the insertion and optimization of quantization and dequantization operations. We will cover the generation of optimized kernels leveraging hardware-specific low-precision instructions and discuss strategies for managing mixed-precision computations effectively. The goal is to understand how compilers enable the practical application of low-precision techniques for efficient model deployment.