Quantization is a set of techniques used to reduce the precision of numbers representing a model's weights and/or activations. Instead of using standard 32-bit floating-point ( $FP32$ ) numbers, we map these values to lower-precision formats, most commonly 8-bit integers ( $INT8$ ) but also increasingly lower-bit floating-point formats like $FP8$ . As introduced, the primary motivation is performance improvement: lower-precision arithmetic is faster on supporting hardware, requires less memory bandwidth, consumes less power, and results in smaller model sizes. However, this compression is inherently lossy; the challenge lies in minimizing the impact on model accuracy while maximizing performance gains. This section examines the fundamental concepts governing this process.

The Quantization Mapping

At its core, quantization maps a real number $R$ from a high-precision range $[R_{min}, R_{max}]$ (e.g., observed FP32 values) to a quantized value $Q$ within a lower-precision integer range $[Q_{min}, Q_{max}]$ (e.g., [-128, 127] for signed INT8). This mapping is typically affine, defined by two parameters: the scale ( $S$ ) and the zero-point ( $Z$ ).

Scale ( $S$ ): A positive real number representing the step size of the quantization. It determines the change in the real value for each increment in the quantized integer value. It's calculated based on the chosen range: $S = \frac{R_{max} - R_{min}}{Q_{max} - Q_{min}}$
Zero-Point ( $Z$ ): An integer value in the quantized range $[Q_{min}, Q_{max}]$ that corresponds to the real value 0.0. It ensures that the real value 0.0 can be represented exactly, which is often important, especially for operations involving padding or biases. It's calculated as: $Z = Q_{min} - \text{round}\left(\frac{R_{min}}{S}\right)$ Note that the rounding function ensures $Z$ is an integer. If 0.0 is within the $[R_{min}, R_{max}]$ range and we want it represented perfectly, $Z$ should map precisely to 0.0.

The quantization function maps a real value $R$ to its quantized integer representation $Q$ : $Q = \text{clamp}\left( \text{round}\left(\frac{R}{S}\right) + Z, Q_{min}, Q_{max} \right)$ The round function typically rounds to the nearest integer (with ties broken arbitrarily or consistently, e.g., round-half-to-even). The clamp function ensures the result stays within the target integer range $[Q_{min}, Q_{max}]$ .

The corresponding dequantization function maps the integer $Q$ back to an approximate real value $R'$ : $R' = S \times (Q - Z)$ Crucially, $R' \approx R$ . The difference $R - R'$ is the quantization error, the source of potential accuracy degradation. Minimizing this error across the model is the central goal of quantization techniques.

Mapping Schemes: Symmetric vs. Asymmetric

The choice of the real range $[R_{min}, R_{max}]$ influences the calculation of $S$ and $Z$ and leads to two primary mapping schemes:

Symmetric Quantization

In symmetric quantization, the real value range is centered around zero, meaning $R_{min} = -R_{max}$ . The maximum absolute value, $\max(|R_{min}|, |R_{max}|)$ , determines the range.

For signed integers (e.g., INT8 [-128, 127]): The range is typically chosen to be symmetric, like $[-127, 127]$ , discarding one value (e.g., -128) to maintain symmetry. The real zero (0.0) maps directly to the integer zero (0), making the zero-point $Z=0$ . The scale is calculated as $S = R_{max} / Q_{max}$ . The quantization function simplifies to $Q = \text{clamp}(\text{round}(R/S), Q_{min}, Q_{max})$ .
For unsigned integers (e.g., UINT8 [0, 255]): Symmetry isn't directly applicable, but a simplified approach might fix $R_{min} = 0$ . Then $S = R_{max} / Q_{max}$ and $Z=0$ .

Symmetric quantization is often preferred for weights, which tend to have distributions centered around zero. Its advantage lies in computational efficiency, as the zero-point offset $Z$ doesn't need to be handled during calculations (it's implicitly 0). However, if the actual data distribution is heavily skewed, forcing a symmetric range can lead to larger quantization errors for values near the less-populated end of the distribution.

Asymmetric Quantization

Asymmetric quantization uses the actual observed minimum and maximum values, $R_{min}$ and $R_{max}$ , without forcing symmetry. Both $S$ and $Z$ are calculated as defined earlier. The real value 0.0 maps exactly to the integer $Z$ .

This scheme can represent the input range more accurately, especially for data distributions that are not centered around zero, such as the outputs of activation functions like ReLU (which are always non-negative). The trade-off is increased computational complexity, as the zero-point $Z$ must be accounted for during arithmetic operations (e.g., adjusting results after multiplications).

Comparison of symmetric and asymmetric mapping from a floating-point range to INT8. Symmetric forces the range around 0.0, mapping to integer 0. Asymmetric uses the actual range, mapping 0.0 to the calculated zero-point Z.

Quantization Granularity: Per-Tensor vs. Per-Channel

Another important aspect is the granularity at which the quantization parameters ( $S$ and $Z$ ) are applied:

Per-Tensor Quantization: A single pair of $(S, Z)$ values is calculated and applied to all elements within an entire tensor (e.g., all weights in a convolutional layer, or all elements in an activation tensor). This is the simplest approach, minimizing the overhead of storing and managing quantization parameters. However, if the value distribution varies significantly across different parts of the tensor (e.g., different output channels in a convolution filter), using a single range $[R_{min}, R_{max}]$ can be suboptimal, potentially clipping important values or underutilizing the quantization range for certain subsets of the tensor.
Per-Channel (or Per-Axis) Quantization: Separate $(S, Z)$ pairs are calculated for slices of the tensor along a specific dimension or axis. For convolutional layer weights, this is most commonly done along the output channel axis (axis=0 for weights of shape [output_channels, input_channels, kH, kW]). This allows the quantization range to adapt more closely to the specific distribution of values within each channel, often leading to significantly better accuracy compared to per-tensor quantization, especially for deep convolutional networks. The trade-off is the increased number of $S$ and $Z$ parameters that need to be stored and used during computation. Other granularities like per-group quantization (grouping channels) also exist as intermediate options.

The choice between per-tensor and per-channel often depends on the specific layer type, its sensitivity to quantization noise, and the target hardware's ability to efficiently handle varying scales and zero-points.

Calibration: Determining the Quantization Range

The effectiveness of quantization hinges on selecting the right clipping range $[R_{min}, R_{max}]$ . This range directly determines the scale $S$ and potentially the zero-point $Z$ . If the range is too narrow, values outside it will be clipped, introducing significant error. If the range is too wide, the quantization steps become large (larger $S$ ), reducing precision for values within the range.

The process of determining the optimal range is called calibration. It typically involves:

Instrumentation: Modifying the model or execution environment to observe the runtime statistics (minimum, maximum, histograms) of FP32 weights and activations.
Data Forwarding: Running inference on a small, representative dataset (the "calibration dataset") using the instrumented model. This dataset should ideally reflect the characteristics of the data the model will encounter during deployment.
Statistics Collection: Gathering the observed ranges or distributions for each tensor that needs to be quantized.
Range Determination: Using a specific algorithm to select the final $[R_{min}, R_{max}]$ $[R_{min}, R_{ma x}]$ based on the collected statistics. Common methods include:
- Min/Max: Simply use the observed minimum and maximum values from the calibration run. This is straightforward but highly sensitive to outliers. A single unusually large or small value can drastically expand the range, reducing precision for the bulk of the values.
- Moving Average Min/Max: Smooth the min/max values over multiple calibration batches to reduce sensitivity to outliers in any single batch.
- Percentile: Exclude a small percentage of the lowest and highest values (e.g., 0.01% or 0.1%) to make the range determination more robust to outliers.
- Entropy (KL Divergence): Select the range $[R_{min}, R_{max}]$ that minimizes the Kullback-Leibler (KL) divergence between the original FP32 distribution and the distribution after quantization and dequantization. This method aims to preserve the information content of the original distribution.
- Mean Squared Error (MSE): Choose the range that minimizes the mean squared error between the original FP32 values and their quantized-dequantized equivalents.

Calibration is primarily associated with Post-Training Quantization (PTQ), where a pre-trained FP32 model is converted to a lower-precision format without retraining. In Quantization-Aware Training (QAT), simulated quantization operations are inserted into the model during training, allowing the model to adapt its weights to the reduced precision and effectively learn the optimal ranges implicitly. However, even QAT might benefit from initial range estimates provided by a calibration step.

Focus on INT8

INT8 (8-bit integer) quantization is currently the most widely adopted low-precision format.

Representation: Uses 8 bits to represent integers. Signed INT8 typically covers the range [-128, 127], while unsigned UINT8 covers [0, 255].
Mapping: Can use either symmetric or asymmetric mapping, with parameters determined via calibration or QAT.
Hardware Support: Dedicated INT8 compute instructions are available on most modern CPUs (e.g., AVX-VNNI, ARM Neon dot product), GPUs (e.g., NVIDIA Tensor Cores, AMD Matrix Cores), and specialized AI accelerators (e.g., TPUs, NPUs), offering significant speedups (often 2-4x or more) over FP32 for operations like matrix multiplication and convolution.
Arithmetic: Computations like $Y = WX + B$ are performed using integer arithmetic. If $W$ and $X$ are INT8, the accumulation ( $W \times X$ ) is typically done at higher precision (e.g., INT32) to avoid overflow. The result is then rescaled (requantized) back to INT8 or dequantized to FP32, incorporating the scales ( $S_W$ , $S_X$ , $S_Y$ ) and zero-points ( $Z_W$ , $Z_X$ , $Z_Y$ ) appropriately. The requantization step $R' = S \times (Q - Z)$ involves integer multiplication by a fractional scale $S$ , often implemented using fixed-point arithmetic with bit shifts and additions.

Emerging Formats: FP8

While INT8 offers substantial benefits, research and hardware development are pushing towards even lower precision, particularly with 8-bit floating-point formats (FP8). Unlike INT8, FP8 retains the exponent-mantissa structure of floating-point numbers, potentially offering a better balance between dynamic range and precision for certain applications, including training.

There isn't a single FP8 standard yet; two prominent variants defined by NVIDIA and supported by other industry players are:

E5M2: 1 sign bit, 5 exponent bits, 2 mantissa bits (+1 implicit leading bit). This format has the same dynamic range as FP16 (5 exponent bits) but significantly less precision (only 3 total precision bits vs. 11 in FP16).
E4M3: 1 sign bit, 4 exponent bits, 3 mantissa bits (+1 implicit leading bit). This format has a narrower dynamic range than E5M2 or FP16 (4 exponent bits) but offers slightly more precision (4 total precision bits).

Bit allocation across different numerical formats used in ML. FP formats dedicate bits to exponent and mantissa, determining range and precision. INT8 uses all non-sign bits for magnitude. FP8 variants trade exponent bits (range) for mantissa bits (precision).

Trade-offs of FP8 vs. INT8:

Dynamic Range: FP8, particularly E5M2, can represent a much wider range of values than INT8, which might be advantageous for values with large dynamic ranges, potentially including gradients during training.
Precision: Within its representable range, INT8 offers uniform precision. FP8 has higher precision near zero but lower precision for large magnitude numbers. E4M3 offers slightly better precision than E5M2.
Hardware Complexity: FP arithmetic units are generally more complex than integer units. However, specialized hardware (like NVIDIA Hopper/Blackwell's Transformer Engine or AMD's MI300) implements efficient FP8 operations.
Zero Representation: Both INT8 (with asymmetric quantization) and FP8 can represent zero exactly.
Special Values: FP8 can represent $\pm \infty$ and NaN, unlike standard INT8.

FP8 requires careful management of scaling factors, similar to FP16 mixed-precision training, often adjusted dynamically. Compilers play a significant role in selecting the appropriate FP8 format (E4M3 vs. E5M2) for different tensors and generating efficient code using hardware-specific FP8 instructions.

Understanding these fundamental concepts, mapping schemes, calibration, granularity, and the characteristics of INT8 and FP8, is prerequisite for designing and implementing the compiler and runtime techniques needed to harness the performance benefits of low-precision computation, which we will examine in subsequent sections.