Hardware Constraint Solving Steps

Translating target performance into a physical hardware specification requires isolating the specific bottlenecks of the language model execution lifecycle. The evaluation process is an exercise in balancing four physical constraints: memory capacity, memory bandwidth, compute throughput, and interconnect speed. Instead of guessing how many GPUs a cluster needs, we systematically calculate the hardware floors for each constraint based on the desired performance metrics. This sequential filtering method ensures your infrastructure can support the resident model weights, handle the dynamic memory allocations of the KV cache, and deliver tokens fast enough to meet user expectations.

The constraint solving pipeline dictates mapping high level performance targets directly to physical hardware capabilities.

Step 1: Solving for Total Memory Capacity

The absolute foundation of hardware sizing is ensuring the physical VRAM can hold both the static model weights and the dynamic KV cache for the requested context length and batch size. If a workload exceeds available VRAM, the system will either crash with out-of-memory errors or heavily degrade by paging to system RAM.

To determine the static weight footprint, multiply the total number of parameters by the precision format size.

$Capacity_{weights} = Parameters \times Bytes\_Per\_Parameter$

For a 70-billion parameter model like meta-llama/Meta-Llama-3.1-70B-Instruct loaded in 16-bit precision (FP16 or BF16, which use 2 bytes per parameter), the static footprint is 140 GB. This immediately rules out dual-GPU 24GB consumer setups like RTX 3090s or 4090s, requiring either a system with larger unified memory like an Apple Silicon Mac Studio with 192GB RAM, or an enterprise setup with two 80GB A100 accelerators.

Next, we calculate the dynamic KV cache required to store previous token attention states.

$Memory_{KV} = 2 \times num\_layers \times num\_kv\_heads \times head\_dim \times seq\_length \times bytes\_per\_element \times batch\_size$

For example, sizing meta-llama/Meta-Llama-3.1-8B for an RTX 4090 with 24GB of VRAM. The model has 32 layers, 8 KV heads, and a head dimension of 128. If we want to serve a batch size of 16 users at a maximum sequence length of 8192 tokens in FP16 (2 bytes):

$2 \times 32 \times 8 \times 128 \times 8192 \times 2 \times 16 = 17,179,869,184$ bytes, or roughly 17.1 GB.

The model weights in FP16 take approximately 16 GB. Adding the 17.1 GB KV cache yields a 33.1 GB requirement. This workload cannot run on a single 24GB RTX 4090. To resolve this constraint, you must either shard the model across two GPUs using Tensor Parallelism, quantize the KV cache to FP8 (reducing cache size to 8.5 GB), or limit the maximum sequence length and batch size.

Step 2: Solving for Memory Bandwidth in the Decode Phase

Once the model fits in VRAM, the next constraint is token generation speed. Autoregressive generation in the decode phase is entirely memory-bandwidth bound. The GPU arithmetic logic units must wait while the entire model weight matrix is transferred from HBM into the SRAM execution registers for every single token generated.

To find the theoretical maximum tokens per second for a batch size of 1, we divide the physical memory bandwidth of the hardware by the resident size of the model.

$Tokens\_Per\_Second = \frac{Memory\_Bandwidth}{Total\_Model\_Size\_in\_Bytes}$

Conversely, to calculate the Time-Per-Output-Token (TPOT), you invert the formula.

$TPOT = \frac{Total\_Model\_Size\_in\_Bytes}{Memory\_Bandwidth}$

Let us size a deployment for Qwen/Qwen2.5-32B quantized to 4-bit AWQ. At 0.5 bytes per parameter plus quantization overhead, the model size is approximately 18 GB. If we deploy this on an Apple M2 Ultra with a memory bandwidth of 800 GB/s:

$800 \text{ GB/s} / 18 \text{ GB} = 44.4 \text{ tokens per second}$

If we deploy the same 18 GB model on a single RTX 3090, which offers roughly 936 GB/s of bandwidth:

$936 \text{ GB/s} / 18 \text{ GB} = 52.0 \text{ tokens per second}$

For human reading speeds, a TPOT of 30 to 50 milliseconds (20 to 33 tokens per second) is generally the acceptable threshold. Both hardware platforms easily meet this SLA for a single user. However, as concurrent requests increase, the effective memory bandwidth per request decreases, increasing the TPOT.

Step 3: Solving for Compute Throughput in the Prefill Phase

While token generation is bound by memory speed, the initial prompt processing phase is constrained by pure compute power. Prefilling a prompt requires large matrix multiplications that fully saturate the Tensor Cores of the GPU. This determines the Time To First Token (TTFT).

First, calculate the required floating point operations (FLOPs) for the forward pass of the prefill stage. The general heuristic is two operations (one multiply, one add) per parameter per token.

$Prefill\_FLOPs = 2 \times Parameters \times Prompt\_Tokens$

Then, divide the required FLOPs by the hardware's effective TFLOP throughput. Hardware utilization is rarely perfect, so a utilization factor (typically 0.5 to 0.7) is applied to realistic hardware specifications.

$TTFT = \frac{Prefill\_FLOPs}{Hardware\_TFLOPs \times Utilization\_Factor}$

Assume we are evaluating a single 80GB H100 PCIe accelerator, which provides approximately 756 TeraFLOPs of FP16 compute. We want to process a 4096-token prompt through mistralai/Mixtral-8x7B-Instruct-v0.1. Despite being a Mixture of Experts model, all parameters must be loaded, but only a subset are active per token. For prefill compute heuristics on dense equivalents, we calculate against the active parameter count (roughly 13B active parameters for Mixtral).

Prefill FLOPs: $2 \times 13,000,000,000 \times 4096 = 106.4 \text{ TeraFLOPs}$.

Applying a 50% utilization factor to the H100 gives an effective throughput of 378 TFLOPs.

TTFT: $106.4 / 378 = 0.28 \text{ seconds}$.

This calculates to a rapid 280 millisecond Time To First Token. If this identical prompt is processed on an older RTX 3060 12GB with only 13 effective TFLOPs of FP16 compute (assuming a 50% utilization factor yields 6.5 effective TFLOPs), the TTFT jumps to over 16 seconds, heavily violating standard interactive SLAs.

Code Implementation: Profiling Memory Allocation

To apply these constraints in practice, engineers rely on highly optimized inference engines like vLLM. You can programmatically restrict GPU memory utilization to observe exactly how much space is left for the KV cache after weights are loaded, allowing you to validate your manual calculations.

Initializing the model with a defined GPU memory utilization reserves headroom for CUDA context while allocating the remainder to KV cache blocks:

from vllm import LLM, SamplingParams
import torch

# Define generation constraints
model_id = "meta-llama/Meta-Llama-3.1-8B"
max_context_length = 8192

# Initialize the LLM with strict memory boundaries
llm = LLM(
    model=model_id,
    tensor_parallel_size=1,            # Single GPU deployment
    gpu_memory_utilization=0.85,       # Cap VRAM usage to 85%
    max_model_len=max_context_length,
    enforce_eager=True,                # Disable CUDA graph capturing for raw profiling
    dtype="float16"                    # Force 2-byte precision
)

# Extract and display the memory profiling results calculated by vLLM
num_gpu_blocks = llm.llm_engine.cache_config.num_gpu_blocks
block_size = llm.llm_engine.cache_config.block_size
bytes_per_block = block_size * 2 * 32 * 8 * 128 * 2  # (Tokens * 2 (K+V) * 32 Layers * 8 KV Heads * 128 HeadDim * 2 bytes)

total_kv_cache_allocated_gb = (num_gpu_blocks * bytes_per_block) / (1024**3)

print(f"Total GPU blocks allocated: {num_gpu_blocks}")
print(f"Tokens per block: {block_size}")
print(f"Calculated KV Cache VRAM footprint: {total_kv_cache_allocated_gb:.2f} GB")

# Test a prompt to verify TTFT and TPOT mechanics
prompts = ["Calculate the memory footprint of a multi-head attention layer."]
sampling_params = SamplingParams(temperature=0.0, max_tokens=100)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"\nGenerated Text: {output.outputs[0].text.strip()}")

By systematically defining required memory capacity, decoding bandwidth, and prefill throughput, you eliminate guesswork in capacity planning. This structured approach guarantees you deploy the precise hardware configuration needed to maintain system stability and meet target latency metrics.