Roofline Performance Modeling Primer

When planning hardware infrastructure for language models, you must determine whether your system will be constrained by compute throughput or by data movement from memory to the processor cores. A workload that requires more calculations than the processors can handle is considered compute-bound, while a workload that spends most of its time waiting for data to arrive from VRAM is memory-bandwidth-bound. The Roofline performance model is a standard graphical tool used to visualize these hardware limits. By mapping the execution lifecycle of a transformer model onto a Roofline chart, you can see why the prompt processing phase is compute-bound, and why token generation scales strictly with memory speeds.

The Arithmetic Intensity Equation

The foundation of the Roofline model relies on a single metric called Arithmetic Intensity ($I$). Arithmetic Intensity measures how many floating-point operations (FLOPs) are performed for every byte of data transferred from memory.

$I = \frac{\text{Total FLOPs}}{\text{Total Memory Bytes Read/Written}}$

If an operation loads a large matrix into memory and performs only a few calculations on it, the arithmetic intensity is low. If the operation loads a small matrix and multiplies it against thousands of other cached values, the arithmetic intensity is high.

For language models, the hardware performance is constrained by two physical ceiling limits:

1. Peak Memory Bandwidth: The maximum speed data can move from VRAM to the Streaming Multiprocessors (SMs), measured in GB/s.
2. Peak Compute: The maximum theoretical throughput of the hardware's Tensor Cores, measured in TFLOPs.

The expected, attainable performance for any operation is determined by the lesser of these two limits:

$\text{Performance} = \min(\text{Peak Compute}, \text{Peak Bandwidth} \times I)$

Defining the Hardware Ridge Point

Every piece of physical hardware has a specific "Ridge Point." This point represents the exact arithmetic intensity where the hardware transitions from being bottlenecked by memory bandwidth to being bottlenecked by compute throughput.

$I_{ridge} = \frac{\text{Peak Compute (TFLOPs)} \times 1000}{\text{Peak Bandwidth (GB/s)}}$

Calculating the ridge point provides an immediate heuristic for evaluating accelerators. A high ridge point means the GPU has incredibly fast compute cores but relatively slower VRAM, requiring high batch sizes to keep the compute cores fed.

Evaluating standard accelerator specifications yields their operational ridge points:

def calculate_ridge_point(gpu_name: str, peak_tflops: float, memory_bandwidth_gb: float):
    # Convert TFLOPs to GFLOPs for the numerator to match GB/s in the denominator
    ridge_point = (peak_tflops * 1000) / memory_bandwidth_gb
    print(f"{gpu_name:<20} | {peak_tflops:>7} TFLOPs | {memory_bandwidth_gb:>6} GB/s | {ridge_point:>6.1f} FLOPs/byte")

print("Hardware Target      | Compute         | Memory      | Ridge Point")
print("-" * 65)
calculate_ridge_point("Apple M2 Ultra", 27.0, 800)
calculate_ridge_point("RTX 4090 24GB", 82.6, 1008)
calculate_ridge_point("RTX 6000 Ada 48GB", 91.1, 960)
calculate_ridge_point("A100 SXM 80GB", 312.0, 2039)
calculate_ridge_point("H100 SXM 80GB", 989.0, 3350)

Running this script reveals a significant architectural shift in modern enterprise GPUs. The NVIDIA H100 requires an arithmetic intensity of 295.2 FLOPs/byte to fully saturate its compute capabilities, nearly double the 153.0 FLOPs/byte required by the older A100. This indicates that while the H100 has significantly higher compute throughput, its VRAM speeds did not increase at the same rate, making memory bandwidth optimization increasingly important.

Mapping the LLM Execution Phases

Language model inference operates in two distinct phases that land on completely different sides of the Roofline ridge point. We can plot these phases to understand how they stress the infrastructure.

The diagonal sloped line represents the memory bandwidth limit (Performance = 2039 * I). The horizontal flat line represents the maximum Tensor Core compute limit (312 TFLOPs). The intersection is the 153 FLOPs/byte ridge point.

Decode Phase (Memory Bound)

During the decode phase, the model generates tokens autoregressively. To generate a single token, the hardware must read the entire model weight matrix from VRAM into the SM registers.

For example, hosting meta-llama/Meta-Llama-3.1-8B in FP16 precision. The model has 8 billion parameters. Each parameter in FP16 takes 2 bytes.

• Total data read = $8 \times 10^9 \times 2 = 16 \text{ GB}$
• FLOPs for a forward pass = $2 \times \text{parameters} = 16 \text{ GFLOPs}$

The Arithmetic Intensity is exactly: $I = \frac{16 \text{ GFLOPs}}{16 \text{ GB}} = 1 \text{ FLOP/byte}$

At 1 FLOP/byte, the workload sits on the extreme far left of the Roofline chart. It is heavily bottlenecked by memory bandwidth. You can calculate the absolute maximum generation speed (tokens per second) by dividing the hardware's peak bandwidth by the model size:

• RTX 4090 Decode Speed: $1008 \text{ GB/s} / 16 \text{ GB} = 63 \text{ tokens/s}$
• A100 Decode Speed: $2039 \text{ GB/s} / 16 \text{ GB} = 127 \text{ tokens/s}$

These are hard physical limits for single-batch decoding. No amount of compiler optimization can exceed these numbers unless the memory footprint of the weights is reduced through quantization.

Prefill Phase (Compute Bound)

During the prefill phase, the user submits a prompt that is processed in parallel. If you submit a prompt with 1024 tokens, the model still reads the 16 GB of weights exactly once, but it multiplies those weights against 1024 tokens simultaneously.

• Total data read = $16 \text{ GB}$ (weights, isolating out activation reads for this baseline calculation)
• FLOPs for the forward pass = $2 \times \text{parameters} \times 1024 \text{ tokens} = 16,384 \text{ GFLOPs}$

$I = \frac{16384 \text{ GFLOPs}}{16 \text{ GB}} = 1024 \text{ FLOPs/byte}$

An arithmetic intensity of 1024 is far above the A100's ridge point of 153. This workload operates on the flat horizontal ceiling of the Roofline chart. The VRAM bandwidth is no longer the bottleneck; the processing speed is dictated entirely by the 312 TFLOPs ceiling of the Tensor Cores.

Scaling with Batch Size

Batching is the primary technique infrastructure operators use to shift workloads out of the memory-bound slope and into the highly efficient compute-bound ceiling.

When you increase the batch size ($B$) during the decode phase, you load the model weights once from VRAM, but perform the matrix-vector multiplication for multiple users simultaneously. The arithmetic intensity scales directly with the batch size. For standard FP16 or BF16 weights, the arithmetic intensity is approximately equal to the batch size:

$I \approx B$

If you are running an A100 GPU, you need a batch size of roughly 153 to hit the ridge point. Below a batch size of 153, the GPU's memory bandwidth is the limiting factor. Above a batch size of 153, the GPU's memory bandwidth keeps up effortlessly, and the total system throughput becomes constrained by the Tensor Cores.

When you evaluate large dense models like Qwen/Qwen2.5-32B, you will find that large batch sizes require enormous amounts of KV cache memory. Capacity planning requires sizing the total VRAM footprint to support a batch size large enough to push the arithmetic intensity as close to the hardware's ridge point as possible without causing Out Of Memory (OOM) errors.