Overview of Common Quantized Model Formats

Quantizing a model changes its fundamental representation. It involves storing weights and activations as lower-precision types, such as 8-bit integers (INT8) or 4-bit integers (INT4), instead of the standard 32-bit floating-point numbers (FP32). Simply saving these low-precision weights into a standard file isn't enough. Important metadata associated with the quantization process itself also needs to be stored.

Consider what happens during quantization: we map the original floating-point range to a smaller integer range. This mapping requires parameters such as:

• Scaling Factors: To convert the quantized integer back to an approximate floating-point value during computation.
• Zero-Points: An offset used, especially in asymmetric quantization, to align the integer zero with the floating-point zero.
• Quantization Scheme: Information about whether quantization was symmetric or asymmetric, per-tensor, per-channel, or per-group.
• Data Type: The specific low-precision format used (e.g., INT4, INT8, NF4).

Standard model serialization formats, often designed for FP32 or FP16 models, don't have dedicated structures to efficiently store and retrieve this metadata alongside the quantized weights. Trying to retrofit them can be cumbersome and inefficient.

This leads to the need for specialized quantized model formats. These formats are designed specifically to package low-precision weights together with all necessary quantization parameters and sometimes even model architecture details or configuration settings. The primary goals driving the development of these formats include:

1. Compactness: Reducing the file size on disk compared to storing the original model or even compared to naive storage of quantized weights.
2. Load Efficiency: Enabling fast loading of the model into memory for inference, often involving memory mapping techniques.
3. Execution Efficiency: Structuring the data in a way that aligns well with optimized low-precision computation kernels used by inference engines.
4. Interoperability: Providing a standard way for models quantized with specific tools to be used by compatible inference libraries or hardware backends.

In this chapter, we'll examine some of the most prevalent formats you'll encounter when working with quantized LLMs:

• GGUF (Georgi Gerganov Universal Format): A flexible format primarily associated with the llama.cpp ecosystem. It's designed to be extensible and store rich metadata, supporting various quantization types and model architectures. We will look at its structure and usage in detail later.
• GPTQ Model Conventions: While GPTQ is a quantization algorithm, models quantized using it often follow specific saving conventions, typically storing weights and quantization parameters (scales, zero-points, group sizes, permutation order) in separate files or within standard formats like Hugging Face's safetensors, relying on associated configuration files. Libraries like AutoGPTQ or ExLlama know how to interpret these conventions.
• AWQ Model Conventions: Similar to GPTQ, AWQ (Activation-aware Weight Quantization) is a method. Models quantized with AWQ also need specific parameters saved alongside the weights, often using established serialization formats combined with configuration files understood by supporting libraries.

It's important to distinguish between the quantization algorithm (like PTQ, GPTQ, AWQ, QAT) which defines how the model weights and/or activations are converted to lower precision, and the file format which defines how the resulting quantized model and its metadata are stored on disk. While some formats are closely tied to specific algorithms (like the conventions used with GPTQ), others like GGUF aim for broader applicability.

The following diagram provides an overview of how these components relate:

Flow showing how quantization algorithms produce weights and parameters, which are then stored in specialized file formats for use by inference engines.

Understanding these formats is essential for practical deployment. They bridge the gap between the theoretical process of quantization and the efficient execution of the resulting compact models. The following sections will provide more detailed information on GGUF, the conventions associated with GPTQ, and the AWQ format, along with the tools used to create and interact with them.