Practical Cluster Capacity Planning

Moving from theoretical bandwidth limits to physical cluster sizing requires bridging the gap between product Service Level Agreements (SLAs) and physical GPU capabilities. You are given a target constraint: a specific number of concurrent users, a maximum acceptable Time-To-First-Token (TTFT), and a strict Time-Per-Output-Token (TPOT) limit. To fulfill these metrics, you must specify the exact count, type, and topology of the GPUs required. The sizing process maps these operational constraints directly onto GPU memory capacity and High Bandwidth Memory (HBM) speed.

Translating Decode SLAs to Memory Bandwidth

During the decode phase, LLM generation is fundamentally memory-bandwidth bound. Every single token generated requires streaming the entire active model weight matrix from GPU VRAM through the streaming multiprocessors (SMs). Therefore, your minimum memory bandwidth requirement is strictly dictated by the model size and your TPOT target.

$Bandwidth_{required} = \frac{Parameters \times Bytes\_Per\_Param}{TPOT_{target}}$

If an application requires running meta-llama/Meta-Llama-3.1-70B-Instruct at FP8 precision (1 byte per parameter) with a strict TPOT of 25 milliseconds (0.025 seconds), the underlying hardware must guarantee a minimum sustained memory bandwidth.

$Bandwidth_{required} = \frac{70 \times 10^9 \times 1 \text{ byte}}{0.025 \text{ seconds}} = 2.8 \text{ TB/s}$

A single NVIDIA A100 80GB provides roughly 2.0 TB/s of theoretical bandwidth, which falls short. To meet this SLA, you must distribute the model across at least two A100 GPUs using Tensor Parallelism, pooling their bandwidth to ~4.0 TB/s. Alternatively, upgrading to a single NVIDIA H100 80GB, which delivers 3.35 TB/s, achieves the SLA on a single device.

Accounting for Concurrent KV Cache Memory

Bandwidth dictates generation speed, but total VRAM capacity dictates concurrency. To determine how many users a cluster node can support before queuing requests, you must calculate the exact VRAM footprint of the model weights and subtract it from the total cluster capacity. The remainder forms your KV cache pool.

$VRAM_{available} = (GPUs \times VRAM_{per\_GPU}) \times Utilization\_Factor$

$VRAM_{KV\_Pool} = VRAM_{available} - VRAM_{weights} - VRAM_{overhead}$

For example, deploying Qwen/Qwen2.5-32B on a local workstation featuring two NVIDIA RTX 4090 24GB GPUs (48GB total). Running at FP16 (2 bytes per parameter), the base weights consume roughly 64GB, resulting in an immediate out-of-memory error. By applying INT4 quantization (0.5 bytes per parameter plus block formatting overhead, averaging roughly 0.7 bytes), the weight footprint shrinks to 22.4GB.

Assuming a standard 90% memory utilization target to leave headroom for dynamic allocations, the available VRAM is 43.2GB. Subtracting the 22.4GB weights and a ~2GB PyTorch/CUDA runtime overhead leaves 18.8GB exclusively for the KV cache.

Using the standard KV cache sizing equation for GQA-based models, you can map this 18.8GB pool to a strict concurrent request limit.

$Memory_{KV} = 2 \times num\_layers \times num\_kv\_heads \times head\_dim \times bytes\_per\_element \times seq\_length \times batch\_size$

If a single token for this specific 32B model configuration consumes exactly 32 bytes of KV cache memory, an 18.8GB pool can sustain approximately 587,500 total tokens in flight. If your application guarantees a maximum context sequence length of 4,096 tokens per request, this dual-4090 node can handle a continuous batch size of 143 concurrent requests.

Memory distribution on a 24GB GPU assuming a 0.90 utilization target (21.6GB usable). Lower precision formats exponentially increase the capacity available for concurrent request states.

Enforcing Capacity Limits in Code

Applying quantization and concurrency bounds locks in strict VRAM and batching limits:

from vllm import LLM, SamplingParams

# Initializing Llama-3.1-8B with strict memory capacity constraints
# Targeting a single 24GB GPU (e.g., RTX 3090 / 4090)
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-8B",
    tensor_parallel_size=1,
    quantization="fp8",         
    gpu_memory_utilization=0.90, # Hard reserve 21.6GB total VRAM
    max_model_len=8192,          # Cap individual sequence length
    enforce_eager=False          # Enable CUDA graphs to minimize execution overhead
)

sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=512)
outputs = llm.generate(["Explain continuous batching limits."], sampling_params)

The gpu_memory_utilization parameter maps directly to the $Utilization\_Factor$ from our formula. By setting this to 0.90, the vLLM engine preemptively allocates the full 21.6GB, segments the available space via PagedAttention, and guarantees no out-of-memory crashes occur during runtime execution.

Sizing Mixture of Experts Architectures

Mixture of Experts models introduce a split calculation between total memory footprint and memory bandwidth limits. mistralai/Mixtral-8x7B-Instruct-v0.1 contains roughly 47 billion total parameters. At FP16 precision, this model requires ~94GB of VRAM just to reside in memory.

However, during inference, the router network only activates 2 experts per token, representing roughly 13 billion parameters. Your cluster must still physically accommodate the full 94GB footprint. You need an Apple Silicon machine with 128GB unified memory, two 48GB workstation GPUs, or two A100 80GB accelerators. But when calculating the expected decode latency, you apply the TPOT bandwidth formula against the 13B active parameters, not the 47B total.

$Bandwidth_{MoE\_Decode} = \frac{Active\_Params \times Bytes\_Per\_Param}{TPOT_{target}}$

This structural advantage allows MoE architectures to deliver remarkably fast token generation speeds relative to their massive parameter scale, making them highly efficient for SLA-constrained production environments.

Multi-Node Topology Scaling

When incoming traffic (QPS) exceeds the concurrent request capacity of a single node, you must scale horizontally. You calculate the total number of hardware nodes required by dividing your projected peak traffic by the node capacity limit.

$Total\_Nodes = \lceil \frac{Peak\_QPS \times Average\_Request\_Duration}{Node\_Concurrency\_Limit} \rceil$

If a single dual-4090 node processes 143 concurrent requests, and your telemetry expects 500 concurrent sessions at peak utilization, you require $\lceil 500 / 143 \rceil = 4$ identical dual-GPU nodes.

A horizontally scaled production cluster. Each node operates independently using Tensor Parallelism (TP) to distribute weight memory internally across GPUs, while the load balancer routes incoming requests to the node with the highest available KV cache pool.