GPU Memory Hierarchy and HBM Bandwidth

When a Large Language Model generates text, the speed of token production is rarely limited by pure computational power. Instead, performance depends entirely on how fast data travels from memory chips to the processing cores. Understanding the physical layout of GPU memory and the strict limits of High Bandwidth Memory (HBM) is the first step in sizing infrastructure for production inference. The hardware architecture determines how fast models run, how many concurrent users you can support, and whether a deployment requires enterprise accelerators or consumer grade hardware.

The Physical Memory Tiers

GPU hardware organizes memory into a hierarchical structure based on proximity to the compute cores, or Streaming Multiprocessors (SMs). As data moves further from the compute cores, storage capacity increases but data transfer speeds drop significantly.

SRAM (Registers and Shared Memory): Located directly inside the compute cores. This tier is incredibly fast, operating at tens of terabytes per second, but has extremely limited capacity. A modern GPU typically holds less than 50 MB of SRAM across the entire die. It is used for immediate matrix calculations.

L2 Cache: Shared across all compute cores on the GPU. It provides several terabytes per second of bandwidth and acts as a buffer between the cores and main VRAM. An NVIDIA H100 provides 50 MB of L2 cache.

High Bandwidth Memory (HBM) and GDDR (VRAM): This is the primary storage area holding your model weights and the runtime KV cache. This tier defines the hardware's usability for LLMs. Enterprise accelerators like the A100 or H100 use stacked HBM positioned immediately adjacent to the GPU die, offering up to 3.35 TB/s on an H100. Consumer cards like the RTX 4090 24GB use GDDR6X, which trades proximity for cost, yielding about 1.0 TB/s. Apple Silicon structures unified memory directly alongside the CPU and GPU, offering up to 800 GB/s on an M2 Ultra with 128GB capacity.

Host System RAM (PCIe): The motherboard system RAM connected to the GPU via PCIe lanes. Bandwidth here drops severely to 32 or 64 GB/s. Offloading model layers to this tier results in severe text generation latency penalties.

Memory hierarchy tiers organized by distance from the compute cores, showing the inverse relationship between capacity and bandwidth.

Memory Bandwidth and Autoregressive Decoding

In autoregressive decoding, generating every single new token requires the hardware to read the entire active model from VRAM into the compute cores. The arithmetic intensity, which is the ratio of compute operations to memory bytes accessed, is approximately 1 FLOP per byte during this phase. Because modern GPUs can perform hundreds of teraflops per second but only read a few terabytes per second, the memory bus acts as the absolute bottleneck.

To determine the theoretical maximum token generation speed for a single user at batch size 1, divide the memory bandwidth by the size of the model weights in memory.

$Max\_Tokens\_Per\_Second = \frac{Memory\_Bandwidth}{Model\_Size\_In\_Bytes}$

Let us size a local deployment running meta-llama/Meta-Llama-3.1-8B in 16-bit precision (FP16). The model weights occupy roughly 16 GB of memory.

On an RTX 4090 24GB with a GDDR6X bandwidth of 1008 GB/s: $1008 \text{ GB/s} / 16 \text{ GB} = 63 \text{ tokens/second}$

On an NVIDIA A100 80GB with an HBM2e bandwidth of 1935 GB/s: $1935 \text{ GB/s} / 16 \text{ GB} = 120 \text{ tokens/second}$

On an NVIDIA H100 80GB with an HBM3 bandwidth of 3350 GB/s: $3350 \text{ GB/s} / 16 \text{ GB} = 209 \text{ tokens/second}$

Notice how the compute capacity (TFLOPs) of these cards scales much faster than their memory bandwidth, but the maximum throughput is governed by the memory bus. These limits represent a hardware ceiling before accounting for dynamic KV cache reads or framework overhead.

Capacity Versus Bandwidth in Production

When sizing setups across enterprise HBM cards, consumer GDDR rigs, or unified memory architectures, you must balance VRAM capacity against memory bandwidth.

If you plan to serve Qwen/Qwen2.5-32B in 16-bit precision, the model weights alone require approximately 64 GB of VRAM.

A single A100 80GB card can fit the model and the dynamic KV cache in its HBM2e memory, delivering inference at roughly 30 tokens per second (1935 GB/s bandwidth divided by 64 GB).

Three RTX 3090 24GB cards provide 72GB of total VRAM, which can hold the model. However, they distribute the memory access over a PCIe or NVLink interconnect. While the aggregate VRAM capacity is sufficient, the effective bandwidth is bottlenecked by the interconnect communication speed and the lowest-performing card in the pipeline, heavily reducing the tokens per second compared to a single HBM card.

An Apple Silicon Mac Studio with 128GB of Unified Memory can easily fit the model but is constrained by its 800 GB/s memory bandwidth, yielding around 12 tokens per second (800 GB/s divided by 64 GB).

Profiling Hardware Utilization

The calculated bandwidth numbers represent maximum theoretical limits. The gap between theoretical limits and actual hardware performance is consumed by kernel launch overhead, memory fragmentation, and framework inefficiencies. You can measure the memory bandwidth your model achieves on-device using native CUDA timing events in PyTorch.

import torch
from transformers import AutoModelForCausalLM

# Load an 8B model in 16-bit precision onto a single GPU
model_id = "meta-llama/Meta-Llama-3.1-8B"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="cuda:0"
)

# Dummy input for a single token decode step
input_ids = torch.randint(0, 32000, (1, 1)).cuda()

# Setup CUDA timing events
start_event = torch.cuda.Event(enable_timing=True)
end_event = torch.cuda.Event(enable_timing=True)

# Warmup passes to initialize CUDA context
for _ in range(3):
    _ = model(input_ids)

# Measure execution time of a single forward pass
start_event.record()
with torch.no_grad():
    _ = model(input_ids)
end_event.record()
torch.cuda.synchronize()

# Calculate time in milliseconds
elapsed_time_ms = start_event.elapsed_time(end_event)

# Calculate effective bandwidth utilization
# Meta-Llama-3.1-8B has ~8.03 billion parameters. In FP16 (2 bytes per param), it is ~16.06 GB.
model_size_gb = 16.06
elapsed_time_s = elapsed_time_ms / 1000
achieved_bandwidth_gb_s = model_size_gb / elapsed_time_s

print(f"Forward Pass Time: {elapsed_time_ms:.2f} ms")
print(f"Achieved Memory Bandwidth: {achieved_bandwidth_gb_s:.2f} GB/s")

Optimizing this gap requires kernel fusion architectures, such as FlashAttention, and custom serving engines, such as vLLM, which structure memory access to keep the HBM saturated constantly.

As you increase the batch size, you read the model weights once from HBM but apply them to multiple input sequences simultaneously. This increases the arithmetic intensity. Eventually, at very high batch sizes, the compute cores saturate and the workload transitions to being compute-bound. In practical deployments, this crossover point determines how you map required user concurrency targets to physical hardware footprints.