Home Blog Learn

All Courses

Practical Quantization for Large Language Models

Chapter 1: Foundations of Model Quantization

Introduction to Model Compression

Why Quantize Large Language Models?

Representing Numbers: Floating-Point vs. Fixed-Point

Integer Data Types in Quantization

Quantization Schemes: Symmetric vs. Asymmetric

Quantization Granularity Options

Measuring Quantization Error

Overview of Quantization Techniques

Chapter 2: Post-Training Quantization (PTQ)

Principles of Post-Training Quantization

Calibration: Selecting Representative Data

Static vs. Dynamic Quantization

Common PTQ Algorithms

Handling Outliers in PTQ

Applying PTQ to LLM Layers

Limitations of Basic PTQ

Hands-on Practical: Applying Static PTQ

Chapter 3: Advanced PTQ Techniques

Introduction to GPTQ

Understanding GPTQ Algorithm Mechanics

AWQ: Activation-aware Weight Quantization

SmoothQuant: Mitigating Activation Outliers

Comparing Advanced PTQ Methods

Implementation Considerations for Advanced PTQ

Hands-on Practical: Quantizing with GPTQ

Chapter 4: Quantization-Aware Training (QAT)

Need for Quantization-Aware Training

Simulating Quantization Effects During Training

Straight-Through Estimator (STE)

Implementing QAT with Deep Learning Frameworks

Fine-tuning Models with Quantization Nodes

Benefits and Drawbacks of QAT vs. PTQ

Practical Considerations for QAT Execution

Hands-on Practical: Setting up a Simple QAT Run

Chapter 5: Quantization Formats and Tooling

Overview of Common Quantized Model Formats

GGUF: Structure and Usage

GPTQ Format: Library Support and Implementation

AWQ Format Details

Working with Hugging Face Transformers and Optimum

Using bitsandbytes for Quantization

Tools for Model Conversion and Loading

Practice: Converting and Loading Quantized Formats

Chapter 6: Evaluating and Deploying Quantized LLMs

Metrics for Evaluating Quantized Models

Benchmarking Inference Speed and Memory Usage

Hardware Considerations for Quantized Inference

Deployment Strategies for Quantized LLMs

Troubleshooting Common Quantization Issues

Analyzing Accuracy vs. Performance Trade-offs

Practice: Benchmarking a Quantized LLM

Post-Training Quantization (PTQ)

Post-Training Quantization (PTQ) applies quantization directly to a model that has already been trained. This approach avoids the need for computationally expensive retraining, making it an attractive option for reducing model size and improving inference speed. PTQ works by converting the model's weights, and sometimes activations, from high-precision formats like $FP32$ to lower-precision integer types such as $INT8$ or $INT4$ .

This chapter covers the practical aspects of PTQ:

The fundamental principles and workflow.
The role of calibration data in determining quantization ranges.
Distinctions between static and dynamic quantization approaches.
Common algorithms for calculating quantization parameters.
Techniques for managing outlier values during quantization.
Understanding the accuracy limitations inherent in basic PTQ methods.
Implementing static PTQ through a hands-on example.

Sections

2.1 Principles of Post-Training Quantization
2.2 Calibration: Selecting Representative Data
2.3 Static vs. Dynamic Quantization
2.4 Common PTQ Algorithms
2.5 Handling Outliers in PTQ
2.6 Applying PTQ to LLM Layers
2.7 Limitations of Basic PTQ
2.8 Hands-on Practical: Applying Static PTQ

© 2025 ApX Machine Learning

;