Chapter 5: Fine Tuning Memory Mechanics

Training and fine-tuning Large Language Models introduce significant memory overhead beyond inference requirements. In addition to base model weights, GPUs must accommodate backward-pass gradients, forward-pass activations, and multi-state optimizer buffers such as AdamW, which can increase memory demand from 2 bytes per parameter up to 16 to 18 bytes per parameter during full parameter updates.

This chapter breaks down memory sizing for training workloads across full fine-tuning, ZeRO memory sharding stages, and parameter-efficient methods like LoRA and QLoRA. It covers the mathematical formulations needed to size adapter rank parameters, manage activation checkpointing, and configure multi-GPU training clusters.