Understanding Quantization Data Types and Formats

The effectiveness of any quantization technique depends critically on the specific numerical formats used to represent the compressed weights and, sometimes, activations. This aspect becomes particularly important as work progresses into sub-8-bit regimes for LLMs, where standard integer and floating-point types are often insufficient or suboptimal. Understanding the characteristics, advantages, and limitations of these low-bit data types is essential for implementing methods like GPTQ, AWQ, and using hardware acceleration effectively.

This section examines the common and emerging data types used in LLM quantization, focusing on formats below INT8. We'll analyze their structure, representational capabilities, and the trade-offs they entail.

Integer Formats (INT)

Integer formats are simpler, typically involving a scaling factor and sometimes a zero-point to map the quantized integers back to the approximate floating-point range.

INT8

While this course focuses on more advanced techniques, INT8 (8-bit integer) serves as a frequent baseline. Using 8 bits provides $2^8 = 256$ distinct levels. This often offers a good balance between compression and accuracy preservation, especially when using techniques like per-channel scaling. Hardware support for INT8 operations is widespread on both CPUs and GPUs, making it a well-established optimization target. Its representation follows the principles discussed earlier:

• Symmetric: $x \approx q \times s$
• Asymmetric: $x \approx (q - z) \times s$

Where $q$ is the INT8 value, $s$ is the scaling factor, and $z$ is the zero-point (an INT8 value representing the floating-point zero).

INT4

Reducing the bit width further to INT4 dramatically cuts memory requirements by half compared to INT8 and enables significantly faster computations on compatible hardware. However, it provides only $2^4 = 16$ distinct representation levels. This extremely coarse quantization makes preserving accuracy much more challenging.

• Representation: Typically uses asymmetric quantization ($x \approx (q - z) \times s$) to better utilize the limited levels, where $q$ is a 4-bit unsigned integer (0 to 15), $s$ is a floating-point scale, and $z$ is the 4-bit integer zero-point (often 7 or 8).
• Challenges: The small number of levels means the quantization error can be substantial. Algorithms like GPTQ and AWQ are specifically designed to mitigate this accuracy loss for INT4 by carefully selecting which weights to quantize and how, often using calibration data.
• Hardware: While direct hardware support for INT4 matrix multiplication exists on newer architectures (like NVIDIA's Hopper and Ada Lovelace with sparsity support), efficient implementation often relies on specialized kernels that pack multiple INT4 values into larger registers (e.g., INT32) and perform bitwise operations.

Lower Integer Formats (INT3, INT2)

Research looks at even lower bit widths like INT3 ($2^3 = 8$ levels) and INT2 ($2^4 = 4$ levels). These offer maximum compression but face severe accuracy degradation. They are currently less common in practical deployments and often require sophisticated techniques like quantization-aware training or highly specialized algorithms to be viable, often restricted to specific model parts.

Low-Bit Floating-Point Formats (FP)

Floating-point formats allocate bits differently, assigning some to an exponent and others to a mantissa (or significand). This allows them to represent a wider dynamic range compared to integers of the same bit width, albeit with varying precision across that range.

FP8 (E5M2 and E4M3)

FP8 utilizes 8 bits, similar to INT8, but adopts a floating-point representation. Two main variants exist, defined by the number of exponent (E) and mantissa (M) bits, plus one sign bit:

• E5M2: 5 exponent bits, 2 mantissa bits. Offers a wider dynamic range (similar to FP16) but lower precision. It's generally better for representing gradients during training or activations which can have large variations.
• E4M3: 4 exponent bits, 3 mantissa bits. Provides a narrower dynamic range but higher precision than E5M2, making it often more suitable for representing weights. It aligns well with the typical distribution of weights in pre-trained models.

FP8 formats strike a balance, offering better handling of outlier values compared to INT8 due to the exponent bits, while still providing significant memory and computational savings over FP16/BF16. Native FP8 support is a feature of recent hardware like NVIDIA H100/L40 GPUs, making it an increasingly important format for efficient LLM inference.

FP4 (e.g., E2M1)

FP4 pushes the envelope with only 4 bits in a floating-point structure. A common configuration is E2M1 (1 sign, 2 exponent, 1 mantissa).

• Representation: Provides extremely limited range and precision. For E2M1, there are only 8 unique representable values (plus sign). Representable values are often clustered non-uniformly.
• Use Cases: FP4 is highly aggressive. It's often explored in research or applied selectively, perhaps combined with block-wise scaling factors or for parameters known to be less sensitive. Achieving acceptable accuracy typically requires specialized quantization schemes or QAT.

Specialized Formats

Recognizing the limitations of standard formats, researchers have developed specialized data types tailored to the observed distributions of values in LLMs.

NormalFloat (NF4)

NormalFloat 4-bit (NF4) is a non-uniform data type introduced with the QLoRA method. It's designed based on the observation that weights in pre-trained LLMs often follow a normal distribution $N(\mu, \sigma^2)$, typically centered around zero ($\mu=0$).

• Core Idea: Instead of uniformly spaced quantization levels like INT4, NF4 defines its representation levels based on the quantiles of a standard normal distribution $N(0, 1)$. It aims to create $2^4 = 16$ levels such that each level represents an equal portion of the probability mass of the $N(0, 1)$ distribution.
• Benefit: This provides higher precision around the mean (zero), where most weights are concentrated, and lower precision further out in the tails. This mapping is considered information-theoretically optimal for normally distributed data.
• Implementation: NF4 requires a single scaling factor to map the normalized $N(0, 1)$ quantiles to the actual range of the weights being quantized. It has demonstrated strong empirical results, particularly when combined with techniques like Double Quantization (quantizing the quantization parameters themselves) as used in QLoRA.

Comparison of representable values for INT4 (uniform spacing), NF4 (non-uniform, denser near zero), and a potential FP4 E2M1 (non-uniform, limited values) when scaled to the range [-1, 1]. This illustrates the different precision distributions offered by each format.

Comparison and Trade-offs

The choice of data type involves navigating several trade-offs:

Feature	INT8	INT4	FP8 (E4M3)	FP4 (E2M1)	NF4
Bits	8	4	8	4	4
Levels	256	16	~240 unique	~14 unique	16
Type	Integer	Integer	Floating Point	Floating Point	Fixed (Non-Uniform)
Uniformity	Yes	Yes	No	No	No
Precision	Moderate	Low	Moderate (Varies)	Very Low (Varies)	High near zero
Range	Limited (by scale)	Very Limited	Wider	Limited	Limited (by scale)
Hardware	Common	Emerging/Kernels	Emerging (GPU)	Research/Kernels	Kernels
Common Use	Weights/Activations	Weights (PTQ)	Weights/Activations	Weights (Research)	Weights (QLoRA)

Factors in Choosing a Format:

1. Target Hardware: Does the CPU or GPU have native support for efficient operations in the target format (e.g., INT8, FP8, INT4)? If not, efficient kernels might be needed, potentially impacting performance or requiring specific libraries (like bitsandbytes).
2. Quantization Algorithm: Some algorithms are inherently tied to certain formats (e.g., QLoRA uses NF4). PTQ methods like GPTQ/AWQ often target INT4.
3. Accuracy Requirements: More aggressive formats (INT4, FP4, NF4) increase the risk of accuracy degradation. The acceptable level of accuracy loss dictates how low you can go.
4. Model Sensitivity: Different parts of an LLM exhibit varying sensitivity to quantization. Mixed-precision approaches might use INT8 or FP8 for more sensitive layers (like attention outputs) and INT4 for others (like linear layers).
5. Distribution of Values: If weights are known to be normally distributed, NF4 might be optimal. If activations have occasional large outliers, an FP format (like FP8) might handle the range better than INT8.

In summary, the data type is not just a detail but a fundamental choice in LLM quantization. Formats like INT4, FP8, and NF4 enable the significant memory and computational savings required for deploying large models efficiently. However, they must be used thoughtfully, often in conjunction with sophisticated quantization algorithms and careful evaluation, to balance performance gains against potential accuracy impacts. Understanding these formats provides the basis for selecting and implementing the advanced techniques discussed in the following chapters.