Decode Phase Memory Bandwidth Constraints

Autoregressive generation dictates that Large Language Models produce text one token at a time. While processing the initial user prompt utilizes the massive parallel computing power of a GPU, generating the response shifts the system into a completely different execution state. During the decode phase, the computational units sit idle for the majority of the time. The speed limit of your text generation is dictated almost entirely by how fast your hardware can move data from High Bandwidth Memory (HBM) into the streaming multiprocessors. Understanding this memory bandwidth constraint is the foundation for estimating inference speeds and sizing hardware for production serving.

Arithmetic Intensity and the Memory Wall

In the decode phase, the generation of a single new token requires the GPU to load the entire set of model weights and the entire historical KV cache for the active sequence. This operation is known as a General Matrix-Vector Multiplication (GEMV).

Arithmetic intensity measures the ratio of compute operations to memory bytes transferred. For a model loaded in 16-bit precision (FP16 or BF16), each parameter requires 2 bytes of memory and performs exactly 2 floating-point operations (one multiply, one add) per token. This yields an arithmetic intensity of exactly 1 FLOP per byte.

Modern AI accelerators are designed for massive arithmetic intensity. An NVIDIA A100 GPU boasts 312 TeraFLOPs of compute but only 2,000 GB/s of memory bandwidth. To keep the compute cores fully saturated, an operation needs an arithmetic intensity of roughly 156 FLOPs per byte. Because decode phase generation operates at 1 FLOP per byte, the compute cores spend over 99 percent of their clock cycles waiting for memory reads to complete.

Calculating Time Per Output Token

Since compute speed is effectively irrelevant during single-batch decoding, we calculate the Time-Per-Output-Token (TPOT) using a direct ratio of memory footprint to memory bandwidth.

$TPOT \approx \frac{Memory_{Weights} + Memory_{KVCache}}{HBM_{Bandwidth}}$

For example, in a local deployment of meta-llama/Meta-Llama-3.1-8B running in FP16 precision on a single NVIDIA RTX 4090 (24GB VRAM). The RTX 4090 has a memory bandwidth of 1,008 GB/s. The memory footprint for 8 billion parameters in 16-bit precision is 16 GB. For a short context window, we can temporarily treat the KV cache size as negligible.

$TPOT = \frac{16 \text{ GB}}{1008 \text{ GB/s}} = 0.0158 \text{ seconds}$

Inverting this latency gives us the maximum theoretical generation speed of 63 tokens per second. No matter how much you optimize your PyTorch code or kernel launches, a single RTX 4090 cannot physically generate text faster than this limit for this specific model precision.

If you load the exact same 8B model onto an enterprise A100 80GB GPU with 2,000 GB/s of bandwidth, the latency drops to 0.008 seconds, yielding 125 tokens per second. The generation speed doubles solely because the VRAM bus is twice as wide.

Let us visualize this relationship across common hardware setups using a larger model. We will map the maximum tokens per second for meta-llama/Meta-Llama-3.1-70B-Instruct quantized to 4-bit precision (INT4), which occupies roughly 35 GB of memory.

The theoretical upper bound for autoregressive generation speed is strictly proportional to hardware memory bandwidth.

Amortizing Bandwidth Constraints with Batching

Operating at a batch size of 1 is highly inefficient for production serving. When multiple concurrent user requests are batched together, the GPU can load the model weights from HBM into SRAM once and use those same weights to compute the next token for all requests in the batch simultaneously.

This shifts the workload from a memory-bound GEMV operation to a more compute-intensive General Matrix-Matrix Multiplication (GEMM). While the weight loading cost is fixed, the memory cost of the KV cache scales linearly with the batch size.

$TPOT = \frac{Memory_{Weights} + (Memory_{KVCache\_Per\_Request} \times Batch\_Size)}{HBM_{Bandwidth}}$

Assume you are serving mistralai/Mixtral-8x7B-Instruct-v0.1 in FP8 on a dual-GPU system containing two RTX 3090s. While Mixtral is a 47 billion parameter model, its Mixture of Experts (MoE) architecture means only 13 billion active parameters are executed per token. However, memory bandwidth depends on data movement. If batch sizes are large enough, tokens will route to every expert, forcing the GPU to load all 47B parameters (47 GB in FP8) from memory anyway.

If each request holds a 2048-token KV cache (occupying roughly 250 MB per request), serving a batch of 64 requests requires 16 GB of VRAM just for the KV cache. The system must now read 47 GB of weights plus 16 GB of KV cache (63 GB total) across the two GPUs. If the effective interconnect bandwidth is sufficient, the system splits this read over the combined 1,872 GB/s bandwidth of the two RTX 3090s. The decode step takes roughly 33 milliseconds, allowing the system to output 30 tokens per second per user, or an aggregate throughput of 1,920 tokens per second across the entire batch.

Measuring Bandwidth Saturation in Code

Forcing a large output sequence with a minimal prompt isolates execution time to the memory-bound decode phase, making it straightforward to benchmark bandwidth throughput:

from vllm import LLM, SamplingParams
import time

# Initialize a 7B model in 16-bit precision (Requires ~14GB VRAM)
llm = LLM(
    model="Qwen/Qwen2.5-7B",
    dtype="float16",
    enforce_eager=True, # Disable CUDA graphs to see raw execution cost
    gpu_memory_utilization=0.9
)

# Force the model to generate exactly 512 tokens
sampling_params = SamplingParams(
    temperature=0.0,
    min_tokens=512,
    max_tokens=512,
    ignore_eos=True
)

# Use a tiny prompt to minimize prefill compute time
prompts = ["Calculate the sequence:"]

start_time = time.perf_counter()
outputs = llm.generate(prompts, sampling_params)
end_time = time.perf_counter()

duration = end_time - start_time
total_tokens = sum(len(out.outputs[0].token_ids) for out in outputs)
tps = total_tokens / duration

print(f"Generated {total_tokens} tokens in {duration:.2f} seconds.")
print(f"Throughput: {tps:.2f} tokens/second.")

If you run this code on a 24GB RTX 3090 (936 GB/s), you will observe a ceiling around 60 to 65 tokens per second. The 14 GB of model weights divided by 936 GB/s results in a theoretical latency of 14.9 ms per token (67 tokens per second). The minor discrepancy between the theoretical calculation and the script output represents the software overhead of the Python runtime, PyTorch tensor dispatching, and minor VRAM fragmentation, proving that physical memory bandwidth dictates your upper performance bounds.