Know how your model performs
before you spin up the cluster.
Model VRAM allocation, KV cache growth, TTFT, and per-user throughput before provisioning compute. Predictions validated against GPU benchmarks.
TTFT Under Concurrency
Llama-3-70B FP8 · 2x H100 · 32k context
Operating Point
4 users · ~4.1s TTFT
Safe Operating Knee
~6 users
Insufficient VRAM
>8 users
Open weights are free. GPU time isn't.
Container pull
→
CUDA-matched wheels
→
Download 140 GB weights
→
Out of Memory
The weights fit. The KV cache does not.
At production context length and concurrency, KV cache memory scales linearly and exhausts VRAM. The out-of-memory crash arrives only after weights are loaded and traffic hits.
The batch size 1 benchmark looked fine.
Under concurrency, per-user tokens per second is a fraction of what a single-stream test showed. Queue times dominate as batch capacity saturates.
Two GPUs were supposed to be twice as fast.
Without NVLink, inter-GPU communication dominates. Tensor-parallel execution stalls waiting on PCIe bus bandwidth.
Get a projection before the bill.
Eliminate trial-and-error cluster sizing. Calculate what fits in memory, project hourly spend, evaluate concurrency limits, and verify whether adding a GPU increases throughput or only adds interconnect overhead.
Memory breakdown
· Llama-3-70B INT4 · 8,192 context · 4 concurrent users
Weights: 35.8 GB
KV Cache: 10.7 GB
Overhead: 5.6 GB
68.7%
20.6%
10.7%
Required VRAM
52.1 GB
Minimum GPU
2x A100 (40GB)
Max batch size
28
Est. throughput
~63 tok/s/user
Results worth checking before sizing a fleet.
Aggregate throughput saturates. Per-user speed does not stop falling.
Past the saturation batch, aggregate throughput flatlines while per-user latency spikes. The crossing point determines the optimal serving decision.
INT4 quantization quarters the batch size at which this throughput saturates, increasing serving capacity. Verify →
Weight quantization does nothing for the KV cache.
INT4 reduces base model weights, but KV cache remains uncompressed by default. At 32k or 128k context lengths, KV cache memory quickly overtakes weight memory.
KV cache quantization reduces memory footprint per user, unlocking larger concurrent batch sizes. Verify →
More GPUs do not mean more speed. Interconnect limits scaling.
Scaling tensor parallelism across PCIe links introduces all-reduce latency that can offset additional compute.
Compare NVLink vs PCIe throughput scaling curves for tensor-parallel deployments. Verify →
Validated against physical hardware benchmarks.
Predictions are verified against a regression suite of physical benchmark runs across hardware architectures, quantizations, and concurrency levels. We track and publish discrepancies transparently.
Verify the numbers before provisioning an 8x or 16x GPU node.
Used by engineers and researchers at leading technology companies and universities worldwide
Architectural specifications behind the numbers.
Structured parameters for open-weight models: attention mechanisms, hidden dimensions, layer counts, context windows, and hardware requirements across quantization formats.
Understand the mechanics behind the metrics.
A 30-chapter masterclass on building large language models from scratch, paired with courses on transformer architecture, attention mechanisms, weight quantization, and serving systems.
Use the calculator to plan your cluster. Use the courses to learn how the hardware and architecture interact.
Need pipeline parallelism, multi-node topologies, or production traffic simulation?
Recent analysis
Sep 8, 2026
Understand how Claude Desktop and Claude Code tool search filtering works
Aug 17, 2026
The exact formulas and techniques to calculate VRAM requirements and predict throughput for massive language models, avoiding costly out-of-memory errors during deployment.
olve ChatGPT action discovery probe errors by handling POST discovery probes with valid JSON-RPC fallback responses