ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Practical Quantization for Large Language Models
Chapter 1: Foundations of Model Quantization
Introduction to Model Compression
Why Quantize Large Language Models?
Representing Numbers: Floating-Point vs. Fixed-Point
Integer Data Types in Quantization
Quantization Schemes: Symmetric vs. Asymmetric
Quantization Granularity Options
Measuring Quantization Error
Overview of Quantization Techniques
Quiz for Chapter 1
Chapter 2: Post-Training Quantization (PTQ)
Principles of Post-Training Quantization
Calibration: Selecting Representative Data
Static vs. Dynamic Quantization
Common PTQ Algorithms
Handling Outliers in PTQ
Applying PTQ to LLM Layers
Limitations of Basic PTQ
Hands-on Practical: Applying Static PTQ
Quiz for Chapter 2
Chapter 3: Advanced PTQ Techniques
Introduction to GPTQ
Understanding GPTQ Algorithm Mechanics
AWQ: Activation-aware Weight Quantization
SmoothQuant: Mitigating Activation Outliers
Comparing Advanced PTQ Methods
Implementation Considerations for Advanced PTQ
Hands-on Practical: Quantizing with GPTQ
Quiz for Chapter 3
Chapter 4: Quantization-Aware Training (QAT)
Need for Quantization-Aware Training
Simulating Quantization Effects During Training
Straight-Through Estimator (STE)
Implementing QAT with Deep Learning Frameworks
Fine-tuning Models with Quantization Nodes
Benefits and Drawbacks of QAT vs. PTQ
Practical Considerations for QAT Execution
Hands-on Practical: Setting up a Simple QAT Run
Quiz for Chapter 4
Chapter 5: Quantization Formats and Tooling
Overview of Common Quantized Model Formats
GGUF: Structure and Usage
GPTQ Format: Library Support and Implementation
AWQ Format Details
Working with Hugging Face Transformers and Optimum
Using bitsandbytes for Quantization
Tools for Model Conversion and Loading
Practice: Converting and Loading Quantized Formats
Quiz for Chapter 5
Chapter 6: Evaluating and Deploying Quantized LLMs
Metrics for Evaluating Quantized Models
Benchmarking Inference Speed and Memory Usage
Hardware Considerations for Quantized Inference
Deployment Strategies for Quantized LLMs
Troubleshooting Common Quantization Issues
Analyzing Accuracy vs. Performance Trade-offs
Practice: Benchmarking a Quantized LLM
Quiz for Chapter 6
Optimizing Large Language Models for Domain Specific Edge Deployment

Static vs. Dynamic Quantization

Post-Training Quantization (PTQ) offers two primary approaches for handling model activations: static quantization and dynamic quantization. The main difference lies in when the quantization parameters (scale factor and zero-point) for the activations are determined and applied. Weights are typically always quantized offline (before inference) in both schemes.

Static Quantization

Static quantization involves quantizing both the model weights and activations before performing inference. Since the range of activations can vary depending on the input data, static quantization requires a calibration step.

  1. Calibration: A representative dataset (calibration dataset) is passed through the model in its original precision (e.g., FP32FP32FP32). The typical range of values (minimum and maximum) for the activations at various points in the model (e.g., outputs of specific layers) is observed and recorded.
  2. Parameter Calculation: Based on the observed ranges during calibration, fixed quantization parameters (scale and zero-point) are calculated for the activations at different quantization points within the model. These parameters are chosen to represent the observed activation distribution reasonably well.
  3. Offline Quantization: Both weights and activations (using the calculated static parameters) are quantized and stored in a lower-precision format like INT8INT8INT8.
  4. Inference: During inference, the model performs computations primarily using integer arithmetic, as both weights and activation quantization parameters are predetermined. The input data is quantized using the parameters determined during calibration for the first layer's input, and intermediate activations are kept in the quantized format between layers where possible.

Static quantization workflow: Calibration determines activation parameters offline, enabling efficient integer-only computations during inference.

Advantages:

  • Higher Potential Performance: Because all quantization parameters are fixed beforehand, the inference can potentially run entirely using highly optimized integer arithmetic instructions on supported hardware (CPUs, GPUs, accelerators), leading to maximum speedup and energy efficiency.
  • Lower Inference Overhead: No runtime cost for calculating activation ranges or quantization parameters.

Disadvantages:

  • Requires Calibration Data: Needs a dataset that accurately reflects the distribution of inputs the model will see in production. Poor calibration data can lead to significant accuracy degradation.
  • Sensitivity to Outliers: If runtime inputs produce activation ranges significantly different from those observed during calibration, accuracy can suffer. Outliers not captured during calibration are clipped, potentially losing important information.

Dynamic Quantization

Dynamic quantization, often referred to as "weight-only quantization" in some contexts, takes a different approach. Weights are quantized offline, similar to static quantization. However, activations are quantized dynamically or on-the-fly during the inference process.

  1. Offline Weight Quantization: Model weights are converted to a lower-precision integer format (e.g., INT8INT8INT8) and stored.
  2. Inference:
    • When input data arrives, the activations are processed layer by layer.
    • For operations involving quantized weights (like matrix multiplications), the incoming floating-point activations are quantized just before the operation. Their range (min/max) is calculated dynamically based on the current batch of data.
    • The computation (e.g., INT8 weight * INT8 activation) is performed.
    • The result is often de-quantized back to floating-point (FP32FP32FP32) before being passed to the next operation or layer that requires floating-point inputs (like certain activation functions or normalization layers).

Dynamic quantization workflow: Weights are quantized offline, but activations are quantized on-the-fly during inference, adding overhead but removing the need for calibration data.

Advantages:

  • No Calibration Data Needed: Simplifies the quantization process as it doesn't require a separate calibration step or dataset for activations.
  • More Effective: Adapts to the specific range of activations for each input, potentially handling unexpected distributions better than static quantization if calibration was suboptimal.
  • Simplicity: Easier to implement initially.

Disadvantages:

  • Higher Inference Overhead: Calculating activation ranges and performing quantization/dequantization dynamically during inference adds computational cost, reducing the potential speedup compared to static quantization.
  • Memory Bandwidth: Continuously reading activations in FP32FP32FP32, quantizing them, performing the compute, and often de-quantizing back can increase memory bandwidth usage compared to a fully static INT8 pipeline.
  • Not Always Fully Integer: The mix of integer weights and dynamically quantized/de-quantized activations means computations might not leverage pure integer arithmetic paths on all hardware, potentially limiting performance gains.

Choosing Between Static and Dynamic Quantization

The choice depends on the specific requirements of your application:

  • Choose Static Quantization if:
    • Maximum inference speed and energy efficiency are primary goals.
    • You have access to representative calibration data.
    • The target hardware has efficient support for full integer arithmetic pipelines.
    • You are willing to invest time in the calibration process and potentially fine-tune it.
  • Choose Dynamic Quantization if:
    • Simplicity and ease of implementation are prioritized.
    • Obtaining good calibration data is difficult or impractical.
    • Some inference overhead is acceptable.
    • The primary goal is weight memory reduction, and activation compute overhead is less critical.

For Large Language Models (LLMs), static quantization is often preferred to maximize throughput and reduce latency due to their computational intensity. However, the large activation ranges and presence of outliers in LLMs make effective calibration particularly important for static quantization. Dynamic quantization can be a simpler starting point, primarily reducing the memory footprint from weights, but the computational overhead of dynamic activation quantization can be substantial for LLMs. Advanced PTQ techniques, discussed later, often build upon the static quantization framework to address its limitations.

  • Quantization for Deep Learning Models, PyTorch Documentation, 2019 (PyTorch Foundation) - Provides practical guidance and implementation details for various quantization types, including static and dynamic quantization, within the PyTorch framework.
  • Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, Dmitry Kalenichenko, 2018 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE) DOI: 10.1109/CVPR.2018.00286 - A foundational paper that introduces methods for quantizing neural networks to enable efficient integer-only inference, laying the technical groundwork for static quantization schemes.
  • Model optimization overview | TensorFlow Lite, TensorFlow Lite Documentation, 2024 (Google) - Offers an overview of different model optimization techniques, including static and dynamic quantization, as applied within the TensorFlow Lite ecosystem.

© 2026 ApX Machine Learning