Chapter 4: Distributed Topologies for LLMs

When a model parameter footprint and KV cache exceed the VRAM of a single accelerator, the workload must be sharded across multiple GPUs. Distributing computation introduces collective communication overhead, making interconnect bandwidth across PCIe, NVLink, and InfiniBand a critical factor in maintaining decoding throughput and minimizing idle GPU bubbles.

This chapter covers distributed execution strategies, including Tensor Parallelism for intra-node matrix sharding, Pipeline Parallelism for inter-node layer distribution, and Expert Parallelism for sparse Mixture of Experts routing. It explores how to balance memory distribution against interconnect bandwidth to design scalable multi-GPU cluster topologies.