ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Practical Quantization for Large Language Models
Chapter 1: Foundations of Model Quantization
Introduction to Model Compression
Why Quantize Large Language Models?
Representing Numbers: Floating-Point vs. Fixed-Point
Integer Data Types in Quantization
Quantization Schemes: Symmetric vs. Asymmetric
Quantization Granularity Options
Measuring Quantization Error
Overview of Quantization Techniques
Quiz for Chapter 1
Chapter 2: Post-Training Quantization (PTQ)
Principles of Post-Training Quantization
Calibration: Selecting Representative Data
Static vs. Dynamic Quantization
Common PTQ Algorithms
Handling Outliers in PTQ
Applying PTQ to LLM Layers
Limitations of Basic PTQ
Hands-on Practical: Applying Static PTQ
Quiz for Chapter 2
Chapter 3: Advanced PTQ Techniques
Introduction to GPTQ
Understanding GPTQ Algorithm Mechanics
AWQ: Activation-aware Weight Quantization
SmoothQuant: Mitigating Activation Outliers
Comparing Advanced PTQ Methods
Implementation Considerations for Advanced PTQ
Hands-on Practical: Quantizing with GPTQ
Quiz for Chapter 3
Chapter 4: Quantization-Aware Training (QAT)
Need for Quantization-Aware Training
Simulating Quantization Effects During Training
Straight-Through Estimator (STE)
Implementing QAT with Deep Learning Frameworks
Fine-tuning Models with Quantization Nodes
Benefits and Drawbacks of QAT vs. PTQ
Practical Considerations for QAT Execution
Hands-on Practical: Setting up a Simple QAT Run
Quiz for Chapter 4
Chapter 5: Quantization Formats and Tooling
Overview of Common Quantized Model Formats
GGUF: Structure and Usage
GPTQ Format: Library Support and Implementation
AWQ Format Details
Working with Hugging Face Transformers and Optimum
Using bitsandbytes for Quantization
Tools for Model Conversion and Loading
Practice: Converting and Loading Quantized Formats
Quiz for Chapter 5
Chapter 6: Evaluating and Deploying Quantized LLMs
Metrics for Evaluating Quantized Models
Benchmarking Inference Speed and Memory Usage
Hardware Considerations for Quantized Inference
Deployment Strategies for Quantized LLMs
Troubleshooting Common Quantization Issues
Analyzing Accuracy vs. Performance Trade-offs
Practice: Benchmarking a Quantized LLM
Quiz for Chapter 6
Optimizing Large Language Models for Domain Specific Edge Deployment

Handling Outliers in PTQ

While Post-Training Quantization aims to map the range of floating-point values to a lower-precision integer range, a significant challenge arises from the presence of outliers. Outliers are values in the weight or activation tensors that lie far outside the typical distribution of values. Even a single extreme value can disproportionately affect the quantization process, potentially leading to substantial accuracy degradation.

The Problem with Outliers

Most basic PTQ algorithms, like MinMax quantization, determine the scaling factor based on the absolute minimum and maximum values observed in a tensor (or a subset during calibration). For example, imagine a tensor where most values are between -1.0 and 1.0, but a single outlier exists with a value of 100.0.

If we use MinMax scaling, the entire range from -1.0 to 100.0 must be mapped to the available integer range (e.g., -128 to 127 for INT8INT8INT8).

scale=max_float−min_floatmax_int−min_int=100.0−(−1.0)127−(−128)=101.0255≈0.396\text{scale} = \frac{\text{max\_float} - \text{min\_float}}{\text{max\_int} - \text{min\_int}} = \frac{100.0 - (-1.0)}{127 - (-128)} = \frac{101.0}{255} \approx 0.396scale=max_int−min_intmax_float−min_float​=127−(−128)100.0−(−1.0)​=255101.0​≈0.396

This large scaling factor means the majority of the original values (between -1.0 and 1.0) will be compressed into a very small portion of the integer range:

  • Original value 1.0 becomes round(1.0/0.396)≈round(2.52)=3round(1.0 / 0.396) \approx round(2.52) = 3round(1.0/0.396)≈round(2.52)=3.
  • Original value -1.0 becomes round(−1.0/0.396)≈round(−2.52)=−3round(-1.0 / 0.396) \approx round(-2.52) = -3round(−1.0/0.396)≈round(−2.52)=−3.

All the detailed information previously represented between -1.0 and 1.0 is now squeezed into roughly 6 integer values (from -3 to 3). The rest of the integer range (from -128 to -4 and 4 to 127) is reserved mostly for mapping the single outlier value of 100.0 (which maps to round(100.0/0.396)≈127round(100.0 / 0.396) \approx 127round(100.0/0.396)≈127, potentially after adjusting for the zero-point). This loss of resolution for the bulk of the values often results in a significant drop in model performance.

The presence of an outlier significantly widens the required range for MinMax quantization, reducing precision for the majority of values.

Strategies for Managing Outliers

Several techniques can mitigate the negative impact of outliers during PTQ:

  1. Clipping (Saturation): Instead of using the absolute minimum and maximum values from the calibration data, we can choose thresholds (clipping values) and force any value outside these thresholds to the boundary value. This process is also known as saturation. For example, we might decide to clip the range based on the 1st and 99th percentiles of the observed values during calibration.

    • Determine a clipping range [cmin,cmax][c_{min}, c_{max}][cmin​,cmax​]. This is often based on percentiles (e.g., 99.9% or 99.99%) of the absolute values observed during calibration.
    • For each value xxx in the tensor: xclipped=max⁡(cmin,min⁡(x,cmax))x_{clipped} = \max(c_{min}, \min(x, c_{max}))xclipped​=max(cmin​,min(x,cmax​)).
    • Calculate the quantization parameters (scale and zero-point) using the clipped minimum (cminc_{min}cmin​) and maximum (cmaxc_{max}cmax​) values.

    Clipping limits the quantization range by saturating outlier values to predefined minimum and maximum thresholds.

    The choice of clipping threshold involves a trade-off: clipping too aggressively discards information from potentially important large values, while clipping too little retains the problem of poor resolution for common values. Selecting the right threshold often requires evaluating the impact on model accuracy using a representative validation dataset.

  2. Alternative Calibration Methods: Algorithms other than simple MinMax, such as those based on minimizing Kullback-Leibler (KL) divergence (Entropy) or using Mean Squared Error (MSE), inherently handle outliers more gracefully. These methods try to find quantization parameters that best preserve the overall distribution shape or minimize the error for the majority of values, rather than being solely dictated by the extreme values. Using percentile-based calibration directly sets the range based on percentiles, effectively clipping the outliers.

  3. Finer Quantization Granularity: As introduced in Chapter 1, quantization can be applied per-tensor, per-channel, or per-group. Outliers are often not uniformly distributed across a large weight tensor. They might be concentrated in specific output channels of a linear layer, for instance.

    • Per-channel quantization: Calculates separate scale and zero-point values for each channel (e.g., each row or column of a weight matrix). An outlier in one channel will only affect the quantization range for that specific channel, leaving others unaffected.
    • Per-group quantization: Calculates parameters for small blocks of weights within a tensor. This offers a compromise between per-tensor and per-channel, isolating outlier impacts to smaller groups.

    Using finer granularity increases the metadata overhead (more scale/zero-point values to store) and might require specific hardware support for optimal performance, but it significantly improves robustness to localized outliers.

  4. Advanced Techniques (Looking Ahead): Methods like SmoothQuant (discussed in Chapter 3) explicitly tackle the issue of activation outliers, which are often more problematic than weight outliers in LLMs. SmoothQuant works by mathematically migrating the quantization difficulty from activations (which have a dynamic range that's hard to quantize) to weights (which are static and easier to quantize) without changing the layer's mathematical function. This often allows for effective INT8 quantization even when activations contain significant outliers.

Choosing the appropriate outlier handling strategy often involves experimentation. Starting with MinMax calibration on a representative dataset, evaluating accuracy, and then trying clipping or finer granularity if accuracy drops significantly is a common workflow. Monitoring the distribution of weights and activations can also provide insights into whether outliers are a likely cause of quantization errors.

© 2026 ApX Machine Learning