Integer Quantization and GGUF Format

Running a 70-billion parameter language model in 16-bit precision requires over 140 GB of VRAM just to store the resident weights. This immediate physical limit prices out standard consumer hardware and confines deployments to multi-node clusters of 80GB enterprise accelerators like the NVIDIA A100 or H100. To run these architectures on accessible hardware, such as a dual-GPU 48GB RTX 4090 setup or an Apple Silicon workstation with 64GB of unified memory, engineers rely on integer quantization. By compressing individual parameters from 16-bit floating-point values down to 4-bit integers, quantization significantly reduces both the memory footprint and the memory bandwidth required to serve the model.

Grouped Integer Quantization Mechanics

Directly casting a 16-bit float (FP16) to a 4-bit integer (INT4) results in unacceptable degradation of model quality because 4 bits can only represent 16 distinct values. Instead, modern formats like AWQ (Activation-aware Weight Quantization) and GPTQ use grouped quantization. The weights are divided into blocks (commonly 128 parameters per block). Each block shares a high-precision scaling factor and a zero-point offset to map the 16 integer values back to the original floating-point distribution.

The relationship during the dequantization step at runtime is:

$W_{FP16} = W_{INT4} \times Scale_{FP16} + Zero_{FP16}$

Because the scale and zero-point values take up additional memory, the actual storage required per parameter is slightly higher than 4 bits. You can calculate the effective bits per weight (bpw) using the block size and the precision of the shared values:

$Effective\_Bits = \frac{Bits_{Weight} \times Block\_Size + Bits_{Scale} + Bits_{Zero}}{Block\_Size}$

Assuming a standard block size of 128, a 16-bit scale, and a 4-bit zero-point offset, the effective bits per weight is:

$Effective\_Bits = \frac{4 \times 128 + 16 + 4}{128} = 4.156 \text{ bits/parameter}$

This equation provides a highly accurate rule of thumb for hardware sizing. For a 32-billion parameter model like Qwen/Qwen2.5-32B, the unquantized FP16 footprint is 65 GB. Applying 4.156 bpw quantization drops the memory requirement dramatically:

$Memory_{GB} = \frac{32,500,000,000 \times 4.156}{8 \times 10^9} = 16.88 \text{ GB}$

This 16.88 GB footprint fits comfortably on a single 24GB consumer GPU like an RTX 3090, leaving roughly 7 GB of VRAM available for the KV cache and PyTorch CUDA context overhead.

The GGUF Format and K-Quants

While formats like AWQ are optimized for GPU Tensor Cores, the GGUF format is engineered for heterogeneous environments. GGUF stores tensors in a binary structure optimized for fast mapping into host system RAM and execution on CPUs or Apple Silicon unified memory architectures, while still allowing specific layers to be offloaded to a discrete GPU.

GGUF relies heavily on mixed-precision quantization matrices known as K-Quants. Different neural network layers exhibit varying sensitivities to precision loss. Attention layers often degrade heavily if compressed too aggressively, whereas large feed-forward network (FFN) layers tolerate high compression. K-Quants mix different bit-depths within the same model file to optimize the performance-to-size ratio.

Common GGUF block formats include:

• Q8_0: Maps every parameter to 8 bits. Provides near-perfect parity with FP16 while halving the memory footprint. Averages 8.5 bpw.
• Q6_K: Uses 6-bit quantization across all tensors. Averages 6.5 bpw.
• Q4_K_M: Uses 6-bit quantization for half the attention and FFN tensors, and 4-bit for the rest. This is the recommended standard for balancing speed, size, and accuracy. Averages 4.8 bpw.
• Q2_K: Aggressive 2-bit quantization for weights, mixed with 4-bit scales. Averages 2.5 bpw, though text generation quality noticeably degrades.

Memory footprint requirements for loading a 70B parameter dense model across different quantization formats.

A meta-llama/Meta-Llama-3.1-70B-Instruct model in Q4_K_M format requires exactly 42 GB of memory. This allows the entire 70B model to load into the unified memory of an Apple M2 Max 64GB machine, or span across two 24GB GPUs using tensor parallelism.

Hardware Bandwidth Limitations

Beyond static memory capacity, quantization heavily dictates the inference speed during the autoregressive decode phase. The decode phase operates at low arithmetic intensity (roughly 1 to 2 floating-point operations per byte loaded). This means generation speed is strictly bound by memory bandwidth, not compute TFLOPs.

You can calculate the theoretical maximum token generation speed for a batch size of 1 using the hardware memory bandwidth:

$Max\_Tokens\_Per\_Second = \frac{Memory\_Bandwidth}{Model\_Size\_in\_Bytes}$

For example, an RTX 3090, which provides 1008 GB/s of VRAM bandwidth. If you were to load a 70B model in FP16 (140 GB), the maximum physical speed the GPU could generate text is $1008 / 140 = 7.2$ tokens per second. By serving the exact same model in a 4-bit quantized format (42 GB), the theoretical ceiling rises to $1008 / 42 = 24.0$ tokens per second. Quantization directly accelerates token generation by reducing the volume of bytes that must travel across the GPU memory bus for every single step.

Serving Quantized Models

To put these formats into production, you select your serving engine based on the quantization target. For GPU-native formats like AWQ, tools like vLLM automatically handle the custom CUDA kernels required for efficient dequantization on the fly.

# Serve an AWQ quantized 32B model on a single 24GB RTX 3090/4090
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-32B-Instruct-AWQ \
    --quantization awq \
    --tensor-parallel-size 1 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.95

For CPU or Apple Silicon environments, GGUF models are executed via llama.cpp or its Python bindings. The engine dynamically maps the binary file into memory and dispatches specific layers to the Metal Performance Shaders (MPS) or CUDA backends if specified by the n_gpu_layers parameter.

from llama_cpp import Llama

# Loading a 4-bit Llama 3.1 8B model into Apple Silicon unified memory
# The weights require approximately 4.9 GB of RAM.
llm = Llama(
    model_path="./Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",
    n_gpu_layers=-1,        # Offload all layers to GPU execution units
    n_ctx=8192,             # Reserve memory for a 8k KV cache sequence
    flash_attn=True         # Enable FlashAttention to optimize dynamic memory
)

output = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a specialized infrastructure planner."},
        {"role": "user", "content": "Calculate the memory for an 8B model at 4-bits."}
    ],
    max_tokens=256
)

print(output["choices"][0]["message"]["content"])