Post-Training Quantization (PTQ) Compilation Flows

Post-Training Quantization (PTQ) offers a pragmatic approach to achieving the performance benefits of low-precision inference without the significant cost and complexity of retraining the model, unlike Quantization-Aware Training (QAT). The core idea is to take a pre-trained floating-point model and convert its weights and activations to lower-precision formats like INT8 or FP8 after training is complete. Compilers play a central role in automating this conversion, transforming the model representation, optimizing the quantized graph, and ultimately generating efficient low-precision code.

The typical PTQ compilation flow involves several distinct stages, transforming the high-level model graph into an optimized, low-precision executable version.

Calibration: Determining Quantization Parameters

The first step in most PTQ flows is calibration. Since we are converting from a continuous floating-point range to a discrete, fixed-point range (e.g., 256 levels for INT8), we need to determine the optimal mapping. This mapping is defined by the scale ($s$) and zero-point ($z$) parameters for each tensor (weights and activations) that needs quantization. The relationship is often affine:

$$\text{float\_value} \approx s \times (\text{quantized\_value} - z)$$

To find appropriate $s$ and $z$ values, the compiler or a dedicated quantization tool analyzes the distribution of values within each tensor. This requires a representative dataset, often called the calibration dataset. The model is run in floating-point mode with inputs from this dataset, and the runtime statistics (ranges, distributions) of weights and intermediate activations are collected.

Common calibration methods include:

1. Min/Max: The simplest method. The scale and zero-point are calculated directly from the minimum and maximum observed floating-point values to span the entire range of the low-precision data type (e.g., $[-128, 127]$ for signed INT8). While easy, this can be sensitive to outliers, potentially underutilizing the quantized range.
2. Mean Squared Error (MSE): Search for scale/zero-point values that minimize the MSE between the original floating-point tensor and its quantized-dequantized equivalent. This is computationally more intensive.
3. Kullback-Leibler (KL) Divergence / Entropy: Treat the floating-point values as a probability distribution and find the quantization parameters that minimize the KL divergence between the original distribution and the distribution after quantization. This often yields better accuracy than simple min/max by focusing on preserving the information content rather than just the range.

Once calibration is complete, the compiler has the necessary $s$ and $z$ parameters for each tensor targeted for quantization.

IR Transformation: Inserting Quantization Operations

With calibration data gathered, the compiler modifies the model's Intermediate Representation (IR). The primary transformation involves inserting quantize and dequantize (often abbreviated as q and dq) operations into the graph.

• A quantize operation takes a floating-point tensor and its learned scale/zero-point, producing a low-precision tensor (e.g., INT8).
• A dequantize operation performs the inverse, converting a low-precision tensor back to floating-point using its associated scale/zero-point.

An original FP32 operation like Conv2D is replaced by a sequence:

Input (FP32) -> quantize -> Input (INT8)
Weights (FP32) -> quantize -> Weights (INT8)

Input (INT8), Weights (INT8) -> Conv2D_INT8 -> Output (INT32 or INT8)

Output (INT32 or INT8) -> dequantize -> Output (FP32)

This initial insertion creates a graph where computations are performed in low precision, but the interfaces between operations might still involve conversions back to FP32. The scales and zero-points determined during calibration are embedded as attributes within these quantize, dequantize, and the new low-precision operator nodes in the IR.

Graph Optimizations for Quantized Models

The naive insertion of Q/DQ nodes often leads to suboptimal performance due to excessive conversions. Therefore, compilers apply specific optimization passes tailored for quantized graphs:

1. QDQ Cancellation: Adjacent dequantize -> quantize pairs that use the same quantization parameters (or compatible ones) are redundant and can be removed. This happens frequently when the output of one quantized layer feeds directly into another.

2. Operator Fusion with Quantization: Standard operator fusion (like merging Conv + Bias + ReLU) needs to be aware of quantization. The goal is to fuse the Q/DQ operations into the main compute kernel.

   • The quantize operation on the input can often be fused into the consumer operation (e.g., the INT8 convolution reads FP32 input and quantizes internally).
   • The dequantize operation on the output can often be fused into the producer operation (e.g., the INT8 convolution writes FP32 output, performing dequantization internally).
   • Alternatively, if subsequent operations also operate in INT8, the fusion might combine multiple INT8 operations directly, avoiding intermediate FP32 conversions entirely. This requires careful management of scales and zero-points ("requantization").

3. Requantization: When the output of one INT8 operation (which might be accumulated in a wider integer type like INT32) needs to be fed into another INT8 operation, a requantization step is necessary. This involves rescaling the intermediate result (e.g., the INT32 accumulator) to the scale and zero-point expected by the next layer's INT8 input. This often involves integer multiplication by a calculated scaling factor (derived from the input, weight, and output scales) followed by a right-shift.

   $$Output_{int8} = \text{round}\left( \frac{s_{in} s_{w}}{s_{out}} \times (\text{Accumulator}_{int32}) \right) + z_{out}$$

   (Note: The exact formula depends on the specific quantization scheme and hardware implementation, often involving fixed-point arithmetic approximations for the scale factor $s_{in}s_{w}/s_{out}$.)

The following diagram illustrates a simplified view of how optimizations might transform the graph:

A view of PTQ graph transformation. Initial insertion of Quantize/Dequantize nodes (left) is followed by optimization passes like fusion (right), where Q/DQ operations are absorbed into the compute kernel.

Lowering and Code Generation

After graph-level optimizations, the compiler lowers the high-level quantized operations (like int8_conv2d) into lower-level IR constructs, potentially including explicit integer arithmetic, shifts for requantization, and vector operations if applicable. This lowered representation is then used by the backend to generate target-specific code, leveraging specialized low-precision hardware instructions if available (e.g., Intel VNNI, ARM NEON dot product instructions, NVIDIA Tensor Core IMMA instructions).

Limitations

While PTQ significantly simplifies the deployment of quantized models, it's not a silver bullet.

• Accuracy: PTQ typically results in a larger accuracy drop compared to QAT, especially for highly sensitive models or very low precisions (below INT8). The lack of retraining prevents the model from adapting its weights to the quantization noise.
• Calibration Data: The quality and representativeness of the calibration dataset are significant for achieving good accuracy.
• Mixed Precision: Often, not all layers are quantized. Sensitivity analysis might reveal that quantizing certain layers leads to unacceptable accuracy loss. Compilers must support generating code for these mixed-precision graphs, efficiently handling transitions between FP32 and INT8 (or other formats).

PTQ compilation flows are essential tools in the ML deployment toolkit, providing a relatively fast path to performance improvements by automating the complex process of model conversion, optimization, and code generation for low-precision execution. Understanding these flows allows engineers to effectively utilize PTQ and diagnose potential issues related to accuracy or performance.