Memory Capacity and Bandwidth Limits

Hardware constraints for large language models generally fall into two distinct physical limitations: the volume of memory you have (capacity) and the speed at which you can read from it (bandwidth). Memory capacity acts as a hard limit that dictates whether a model can run on a specific device, determining the maximum parameter count and context window length. Memory bandwidth acts as a speed governor, determining exactly how fast the hardware can generate text once the model is loaded. Understanding the relationship between these two constraints is required to size infrastructure efficiently and guarantee production latency SLAs.

Memory Capacity: The Hard Limit

GPU Memory Capacity (VRAM) must be large enough to house the entire state of the model and its execution environment simultaneously. If the total memory footprint exceeds the physical VRAM (for example, attempting to load a 140GB model onto a 24GB RTX 4090 or an 80GB A100), the process will fail with an Out-of-Memory (OOM) error, or fallback to system RAM, which incurs a massive throughput penalty.

The required capacity consists of four main components stacked on top of one another:

1. Static Model Weights: The resident parameters of the model.
2. KV Cache: The dynamic memory allocated to store past self-attention states.
3. Forward Activations: Temporary memory buffers used during runtime operations.
4. Framework Overhead: CUDA context overhead and PyTorch/vLLM reserved buffers.

Diagram illustrating the GPU memory stack required to run a transformer model in inference mode.

To calculate the absolute minimum VRAM required just for the static model weights, multiply the parameter count by the number of bytes used per parameter:

$M_{weights} = Parameters \times \text{Bytes per Parameter}$

An 8-billion parameter model like meta-llama/Meta-Llama-3.1-8B running in standard FP16 precision (2 bytes per parameter) requires:

$8,000,000,000 \times 2 \text{ bytes} = 16 \text{ GB}$

Loading this 16 GB model onto a consumer GPU with 24 GB of VRAM, such as an RTX 3090 or RTX 4090, leaves approximately 8 GB of VRAM for the KV cache, activations, and overhead. This easily supports single-batch inference with a large context window.

However, sizing changes drastically for larger architectures. A 70-billion parameter model like meta-llama/Meta-Llama-3.1-70B-Instruct in FP16 requires 140 GB just for the weights. This exceeds the capacity of a single 80 GB enterprise accelerator (like an NVIDIA A100 or H100). To fit this model without offloading, you must either shard the model across two 80 GB GPUs using tensor parallelism, or apply quantization to reduce the bytes per parameter. Quantizing the same 70B model to INT4 (0.5 bytes per parameter, plus overhead) compresses the weights to approximately 39 GB. This quantized version now easily fits onto a single 80 GB A100 or a dual-GPU 48 GB setup (such as two RTX 3090s) with headroom to spare for the KV cache.

Memory Bandwidth: The Autoregressive Governor

While memory capacity determines if the model fits, High Bandwidth Memory (HBM) speed determines how fast the model generates tokens.

During the decode phase of text generation, the process is heavily memory-bandwidth-bound. To generate a single token, the GPU must read every single model weight from VRAM, pass it through the streaming multiprocessors (SMs) for a quick matrix-vector multiplication, and discard it. This operation has an extremely low arithmetic intensity of approximately 1 FLOP per byte. The bottleneck is not how fast the GPU can compute (TFLOPs), but how fast it can feed data to the SMs (GB/s).

You can calculate the absolute physical minimum Time-Per-Output-Token (TPOT) for a single-batch request using the model size and the physical memory bandwidth of the GPU:

$TPOT = \frac{\text{Model Weights Size in GB}}{\text{Memory Bandwidth in GB/s}}$

Using meta-llama/Meta-Llama-3.1-8B (16 GB in FP16), we can evaluate expected latency across different hardware setups:

1. Consumer GPU (NVIDIA RTX 4090): Has a memory bandwidth of 1,008 GB/s. $16 \text{ GB} / 1008 \text{ GB/s} = 0.0158 \text{ seconds (15.8 ms per token)}$ This translates to a maximum theoretical generation speed of ~63 tokens per second.
2. Enterprise Accelerator (NVIDIA H100 SXM): Has a memory bandwidth of 3,350 GB/s. $16 \text{ GB} / 3350 \text{ GB/s} = 0.0047 \text{ seconds (4.7 ms per token)}$ This yields a maximum theoretical speed of ~212 tokens per second.
3. Unified Memory Architecture (Apple M2 Ultra): Has 800 GB/s bandwidth. $16 \text{ GB} / 800 \text{ GB/s} = 0.02 \text{ seconds (20 ms per token)}$ This yields ~50 tokens per second.

These calculations establish the hard physical speed limit for the hardware. Software framework overheads and dynamic KV cache loading will reduce the actual observed throughput by 10% to 20%, but the physical bandwidth dictates the theoretical ceiling.

The Impact of Batching on Bandwidth Utilization

To improve cluster economics, infrastructure operators process multiple requests concurrently. Batching modifies the arithmetic intensity of the decoding phase. Instead of reading the 16 GB model from memory to generate 1 token (batch size 1), continuous batching reads the 16 GB model to generate $B$ tokens simultaneously across $B$ concurrent sequences.

This amortization of the memory read operation causes the aggregate throughput (tokens per second) to scale non-linearly with batch size, eventually reaching a plateau at the GPU memory bandwidth ceiling:

$eff(B) = S \times (1 - e^{-B / S})$

Here, $S$ represents the saturation point where the system transitions from being memory-bandwidth-bound to compute-bound, or until VRAM capacity is entirely consumed by the growing KV cache footprint.

When deploying Mixture of Experts (MoE) architectures like mistralai/Mixtral-8x7B-Instruct-v0.1, high batch sizes introduce an anomaly called expert thrashing. Because tokens in a large batch are routed to different sparse experts simultaneously, the GPU is forced to load a higher percentage of the total resident parameters into active memory. This degrades the bandwidth amortization effect, lowering the saturation threshold $S$ compared to a dense model of equivalent active parameter count.

Practical Hardware Sizing Scenarios

When balancing capacity against bandwidth for production multi-GPU configurations, the target latency SLA dictates the choice.

For example, deploying Qwen/Qwen2.5-32B in 8-bit precision (approximately 32 GB footprint). A single 24 GB GPU lacks the capacity. To fit the model, you can span it across dual 24 GB RTX 3090s using pipeline or tensor parallelism.

from transformers import AutoModelForCausalLM
import torch

# Distributing Qwen2.5-32B across multiple GPUs using 8-bit quantization.
# This reduces the parameter capacity footprint to ~32GB, safely fitting 
# onto a dual 24GB VRAM configuration (48GB total).
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-32B",
    device_map="auto",
    torch_dtype=torch.float16,
    load_in_8bit=True
)

In a dual-GPU tensor-parallel setup, you pool the capacity (24 GB + 24 GB = 48 GB) while theoretically utilizing the aggregate bandwidth of both cards (~936 GB/s + 936 GB/s), minus the latency overhead incurred by the PCIe or NVLink interconnect communicating intermediate activation states between the cards.

Alternatively, you could load the same 32 GB model onto an Apple Silicon Mac Studio with 64 GB or 128 GB of unified memory. The Apple architecture excels in memory capacity, allowing massive models to run entirely on a single node without multi-GPU sharding. However, you are capped at the fixed unified memory bandwidth limit (400 GB/s to 800 GB/s depending on the chip tier), meaning a single user stream will generate tokens significantly slower than a multi-GPU setup with higher aggregate HBM bandwidth.