ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Practical Quantization for Large Language Models
Chapter 1: Foundations of Model Quantization
Introduction to Model Compression
Why Quantize Large Language Models?
Representing Numbers: Floating-Point vs. Fixed-Point
Integer Data Types in Quantization
Quantization Schemes: Symmetric vs. Asymmetric
Quantization Granularity Options
Measuring Quantization Error
Overview of Quantization Techniques
Quiz for Chapter 1
Chapter 2: Post-Training Quantization (PTQ)
Principles of Post-Training Quantization
Calibration: Selecting Representative Data
Static vs. Dynamic Quantization
Common PTQ Algorithms
Handling Outliers in PTQ
Applying PTQ to LLM Layers
Limitations of Basic PTQ
Hands-on Practical: Applying Static PTQ
Quiz for Chapter 2
Chapter 3: Advanced PTQ Techniques
Introduction to GPTQ
Understanding GPTQ Algorithm Mechanics
AWQ: Activation-aware Weight Quantization
SmoothQuant: Mitigating Activation Outliers
Comparing Advanced PTQ Methods
Implementation Considerations for Advanced PTQ
Hands-on Practical: Quantizing with GPTQ
Quiz for Chapter 3
Chapter 4: Quantization-Aware Training (QAT)
Need for Quantization-Aware Training
Simulating Quantization Effects During Training
Straight-Through Estimator (STE)
Implementing QAT with Deep Learning Frameworks
Fine-tuning Models with Quantization Nodes
Benefits and Drawbacks of QAT vs. PTQ
Practical Considerations for QAT Execution
Hands-on Practical: Setting up a Simple QAT Run
Quiz for Chapter 4
Chapter 5: Quantization Formats and Tooling
Overview of Common Quantized Model Formats
GGUF: Structure and Usage
GPTQ Format: Library Support and Implementation
AWQ Format Details
Working with Hugging Face Transformers and Optimum
Using bitsandbytes for Quantization
Tools for Model Conversion and Loading
Practice: Converting and Loading Quantized Formats
Quiz for Chapter 5
Chapter 6: Evaluating and Deploying Quantized LLMs
Metrics for Evaluating Quantized Models
Benchmarking Inference Speed and Memory Usage
Hardware Considerations for Quantized Inference
Deployment Strategies for Quantized LLMs
Troubleshooting Common Quantization Issues
Analyzing Accuracy vs. Performance Trade-offs
Practice: Benchmarking a Quantized LLM
Quiz for Chapter 6
Optimizing Large Language Models for Domain Specific Edge Deployment

Overview of Common Quantized Model Formats

Quantizing a model changes its fundamental representation. It involves storing weights and activations as lower-precision types, such as 8-bit integers (INT8) or 4-bit integers (INT4), instead of the standard 32-bit floating-point numbers (FP32). Simply saving these low-precision weights into a standard file isn't enough. Important metadata associated with the quantization process itself also needs to be stored.

What happens during quantization: we map the original floating-point range to a smaller integer range. This mapping requires parameters such as:

  • Scaling Factors: To convert the quantized integer back to an approximate floating-point value during computation.
  • Zero-Points: An offset used, especially in asymmetric quantization, to align the integer zero with the floating-point zero.
  • Quantization Scheme: Information about whether quantization was symmetric or asymmetric, per-tensor, per-channel, or per-group.
  • Data Type: The specific low-precision format used (e.g., INT4, INT8, NF4).

Standard model serialization formats, often designed for FP32 or FP16 models, don't have dedicated structures to efficiently store and retrieve this metadata alongside the quantized weights. Trying to retrofit them can be cumbersome and inefficient.

This leads to the need for specialized quantized model formats. These formats are designed specifically to package low-precision weights together with all necessary quantization parameters and sometimes even model architecture details or configuration settings. The primary goals driving the development of these formats include:

  1. Compactness: Reducing the file size on disk compared to storing the original model or even compared to naive storage of quantized weights.
  2. Load Efficiency: Enabling fast loading of the model into memory for inference, often involving memory mapping techniques.
  3. Execution Efficiency: Structuring the data in a way that aligns well with optimized low-precision computation kernels used by inference engines.
  4. Interoperability: Providing a standard way for models quantized with specific tools to be used by compatible inference libraries or hardware backends.

In this chapter, we'll examine some of the most prevalent formats you'll encounter when working with quantized LLMs:

  • GGUF (Georgi Gerganov Universal Format): A flexible format primarily associated with the llama.cpp ecosystem. It's designed to be extensible and store rich metadata, supporting various quantization types and model architectures. We will look at its structure and usage in detail later.
  • GPTQ Model Conventions: While GPTQ is a quantization algorithm, models quantized using it often follow specific saving conventions, typically storing weights and quantization parameters (scales, zero-points, group sizes, permutation order) in separate files or within standard formats like Hugging Face's safetensors, relying on associated configuration files. Libraries like AutoGPTQ or ExLlama know how to interpret these conventions.
  • AWQ Model Conventions: Similar to GPTQ, AWQ (Activation-aware Weight Quantization) is a method. Models quantized with AWQ also need specific parameters saved alongside the weights, often using established serialization formats combined with configuration files understood by supporting libraries.

It's important to distinguish between the quantization algorithm (like PTQ, GPTQ, AWQ, QAT) which defines how the model weights and/or activations are converted to lower precision, and the file format which defines how the resulting quantized model and its metadata are stored on disk. While some formats are closely tied to specific algorithms (like the conventions used with GPTQ), others like GGUF aim for broader applicability.

The following diagram provides an overview of how these components relate:

Flow showing how quantization algorithms produce weights and parameters, which are then stored in specialized file formats for use by inference engines.

Understanding these formats is essential for practical deployment. They bridge the gap between the theoretical process of quantization and the efficient execution of the resulting compact models. The following sections will provide more detailed information on GGUF, the conventions associated with GPTQ, and the AWQ format, along with the tools used to create and interact with them.

  • GGUF (Georgi Gerganov Universal Format) Specification, Georgi Gerganov and the llama.cpp Contributors, 2023 - The official technical specification for the GGUF format, a specialized file format widely used for storing quantized Large Language Models (LLMs) and their metadata.
  • GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2023 ICLR 2023 DOI: 10.48550/arXiv.2210.17323 - The original research paper describing the GPTQ quantization algorithm, which underpins the saving conventions for GPTQ-quantized models discussed in the section.
  • AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, Song Han, 2023 MLSys 2024 DOI: 10.48550/arXiv.2306.00978 - The original research paper proposing the AWQ quantization method, outlining the parameters and conventions relevant for its quantized model formats.

© 2026 ApX Machine Learning