Successfully launching a distributed training run verifies correctness, but it does not guarantee efficiency. When scaling across dozens or hundreds of GPUs, minor inefficiencies in synchronization or memory management accumulate, leading to significant degradation in throughput. Performance engineering in this environment shifts from simple code optimization to analyzing system-level interactions between the CPU, GPU kernels, and the network fabric.

This section addresses the diagnostic process for distributed setups. We begin by generating and interpreting trace data using the PyTorch Profiler to visualize the execution timeline. You will learn to distinguish between compute-bound and communication-bound operations, identifying gaps where the GPU sits idle.

The discussion includes specific techniques for analyzing communication overlap. In an ideal FSDP setup, the communication required to gather sharded parameters happens concurrently with the computation of the previous layer. We will examine how to verify this behavior and adjust backward prefetching settings to minimize the exposed communication time.

Memory management also requires close inspection. Even when the total parameters fit within VRAM, memory fragmentation can trigger expensive allocator retries or out-of-memory errors. We will cover methods to inspect the caching allocator's state to resolve these issues. Finally, the chapter moves to high-level metrics, defining Model Flops Utilization (MFU) as the primary indicator for training efficiency. You will learn to calculate the theoretical peak performance of your hardware and measure how close your configuration comes to that limit using the standard utilization formula:

$\text{MFU} = \frac{\text{Achieved TFLOPS}}{\text{Peak Device TFLOPS}}$