Designing and Sizing Production LLM Infrastructure

Sizing and designing hardware infrastructure for Large Language Models requires balancing GPU memory allocation, bandwidth limits, and compute throughput against production latency targets. This curriculum covers practical sizing calculations and heuristics to plan cluster capacities for both inference and fine-tuning workloads.

Topics cover transformer execution lifecycles, static parameter and dynamic runtime memory interactions, precision formats (FP16, FP8, INT4, GGUF), and attention topologies (MHA, GQA, MLA, PagedAttention). It examines how serving dynamics-such as compute-bound prefill and bandwidth-bound decode phases-dictate hardware selection, distributed parallelism strategies, and multi-GPU cluster topologies.

Prerequisites Familiarity with transformer models, Python, and basic PyTorch workflows.

GPU Memory Sizing
Calculate exact GPU memory requirements for model parameters, KV cache, and training states.
Inference Bottleneck Identification
Differentiate between compute-bound prefill phases and memory-bandwidth-bound decode phases.
Distributed System Planning
Design cluster layouts utilizing Tensor, Pipeline, and Expert Parallelism for large models.
Fine-Tuning Resource Allocation
Estimate hardware resources needed for full parameter training versus LoRA and QLoRA.
Capacity Planning
Apply roofline modeling to map required Latency SLAs to physical GPU cluster configurations.