Chapter 2: GPU Memory Precision and Architectures

GPU memory allocation for transformer models depends directly on numeric precision and architecture design. Memory capacity dictates whether a model can load into VRAM, while memory bandwidth governs decoding speed during text generation. Offloading model weights or state to system RAM incurs severe latency penalties, making accurate byte-level footprint calculations essential before provisioning.

This chapter examines the GPU memory hierarchy, standard floating-point representations such as FP16, BF16, and FP8, integer quantization formats like INT4 and GGUF, and Mixture of Experts parameter accounting. It details how quantization and architectural choices reduce resident memory footprints to fit hardware constraints without compromising serving stability.