Model LLM memory, throughput, and latency
before provisioning compute.
Simulate dynamic KV cache growth, Tensor Parallel interconnect bottlenecks, and concurrency latency cliffs with physical benchmark accuracy.
TTFT Under Concurrency
Llama-3-70B FP8 · 2x H100 · 32k context
Operating Point
4 users · ~4.1s TTFT
Safe Operating Knee
~6 users
Insufficient VRAM
>8 users
Open weights are free. GPU time isn't.
Container pull
→
CUDA-matched wheels
→
Download 140 GB weights
→
Out of Memory
The weights fit. The KV cache does not.
At production context length and concurrency, KV cache memory scales linearly and exhausts VRAM. The out-of-memory crash arrives only after weights are loaded and traffic hits.
The batch size 1 benchmark looked fine.
Under concurrency, per-user tokens per second is a fraction of what a single-stream test showed. Queue times dominate as batch capacity saturates.
Two GPUs were supposed to be twice as fast.
Without NVLink, inter-GPU communication dominates. Tensor-parallel execution stalls waiting on PCIe bus bandwidth.
Get a projection before the bill.
Eliminate trial-and-error cluster sizing. Calculate what fits in memory, project hourly spend, evaluate concurrency limits, and verify whether adding a GPU increases throughput or only adds interconnect overhead.
Memory breakdown
· Llama-3-70B FP8 · 32,768 context · 4 concurrent users
Weights: 70.5 GB
Active KV: 21.5 GB
Available Pool: 68.0 GB
44.1%
13.4%
42.5%
Active VRAM
92.0 GB
57.5% of 160 GB on 2x H100
Concurrency Limit
15 concurrent users
68.0 GB dynamic headroom
First-Token Latency (TTFT)
~3.75s
p50: 3.75s · p95: 3.84s
Est. Throughput
~39 tok/s/user
156 tok/s cluster throughput
Results worth checking before sizing a fleet.
Aggregate throughput saturates. Per-user speed does not stop falling.
Past the saturation batch, aggregate throughput flatlines while per-user latency spikes. The crossing point determines the optimal serving decision.
INT4 quantization quarters the batch size at which this throughput saturates, increasing serving capacity. Verify →
Weight quantization does nothing for the KV cache.
INT4 reduces base model weights, but KV cache remains uncompressed by default. At 32k or 128k context lengths, KV cache memory quickly overtakes weight memory.
KV cache quantization reduces memory footprint per user, unlocking larger concurrent batch sizes. Verify →
More GPUs do not mean more speed. Interconnect limits scaling.
Scaling tensor parallelism across PCIe links introduces all-reduce latency that can offset additional compute.
Compare NVLink vs PCIe throughput scaling curves for tensor-parallel deployments. Verify →
Validated against physical hardware benchmarks.
Predictions are verified against a regression suite of physical benchmark runs across hardware architectures, quantizations, and concurrency levels. We track and publish discrepancies transparently.
Verify the numbers before provisioning an 8x or 16x GPU node.
Used by engineers and researchers at leading technology companies and universities worldwide
Architectural specifications behind the numbers.
Structured parameters for open-weight models: attention mechanisms, hidden dimensions, layer counts, context windows, and hardware requirements across quantization formats.
Understand the mechanics behind the metrics.
A 30-chapter masterclass on building large language models, alongside courses on transformer architecture, attention mechanisms, weight quantization, and serving systems.
Explore the underlying model architecture in the courses.
How To Build A Large Language Model
Acquire the engineering skills to construct, train, and optimize sophisticated large language models.
Read Masterclass
Introduction to Transformer Models
Grasp the core architecture and attention mechanisms of Transformer models used in state-of-the-art NLP.
View Course
Practical Quantization for Large Language Models
Implement LLM quantization techniques (PTQ, QAT, GPTQ, GGUF) to reduce model size and improve inference speed.
View Course
Deploying Quantized LLMs for Efficient Inference
Efficiently deploy quantized LLMs on various hardware by mastering advanced techniques and toolkits.
View CourseNeed pipeline parallelism, multi-node topologies, or production traffic simulation?
Recent analysis
Sep 8, 2026
Understand how Claude Desktop and Claude Code tool search filtering works
Aug 17, 2026
The exact formulas and techniques to calculate VRAM requirements and predict throughput for massive language models, avoiding costly out-of-memory errors during deployment.
olve ChatGPT action discovery probe errors by handling POST discovery probes with valid JSON-RPC fallback responses