Know what runs
before you spin up the cluster.
Memory, tokens per second, time to first token, and hourly cost for any open-weight model on physical GPUs. Evaluated against measured runs.
Select Spec
Fits
Model
Llama-3-70B
Quantization
4-bit (INT4)
Preferred GPU
2x H100 (80GB)
Context length
8,192
Concurrent users
4
Total VRAM
52.5 GB
Est. throughput
~91 tok/s/user
Est. TTFT
~1154 ms
Est. hourly cost
$4.58 / hr
Open weights are free. GPU time isn't.
Container pull
→
CUDA-matched wheels
→
Download 140 GB weights
→
Out of Memory
The weights fit. The KV cache does not.
At production context lengths and active concurrency, the cache is what runs out. The out-of-memory error arrives after the model has already loaded.
The batch size 1 benchmark looked fine.
Under concurrency, per-user tokens per second is a fraction of what a single-stream test showed. Queue times dominate as batch capacity saturates.
Two GPUs were supposed to be twice as fast.
Over the wrong interconnect, they are not. High-latency communication dominates, stalling tensor-parallel execution on bus bandwidth bottlenecks.
Get a projection before the bill.
Every provisioning call the setup above would otherwise make you find out the hard way: what fits, what it costs per hour, how it holds up under concurrent traffic, whether the extra GPU is buying speed or just cost. All of it, before anything is rented.
Memory breakdown
Weights: 35.8 GB
KV Cache: 10.7 GB
Overhead: 6.0 GB
68.2%
20.5%
11.4%
Required VRAM
52.5 GB
Minimum GPU
2x A100 (40GB)
Max batch size
28
Est. throughput
~91 tok/s
Results worth checking before sizing a fleet.
Aggregate throughput saturates. Per-user speed does not stop falling.
Past the saturation batch, aggregate throughput flatlines while per-user latency spikes. The crossing point determines the optimal serving decision.
INT4 quantization quarters the batch size at which this throughput saturates, increasing serving capacity. Verify →
Weight quantization does nothing for the KV cache.
At long context, the cache is what runs out. Quantizing your weights to INT4 does not touch the cache.
A 30B/3B MoE model serves like a 3B at batch size 1 but drifts toward a 30B under concurrency. Verify →
More GPUs is not more speed.
Adding compute is sometimes just adding a bottleneck.
Prefill is where unified memory architectures stop paying. Verify →
Evaluated against actual benchmark runs.
Every model prediction is verified against a regression suite of measured runs: published file sizes, benchmark results, and hardware actually used, over hundreds of cases across memory, decode speed, and fine-tuning throughput. We publish all discrepancies transparently.
Verify the math before you provision 8 or 16 GPUs.
Used by engineers and researchers at leading technology companies and universities worldwide
The data the estimates are built on.
Every open-weight model with its architecture, attention structure, layer and head counts, context length, and per-quantization hardware requirements. Community-submitted benchmark runs, with the hardware they were measured on.
The same reference that explains the numbers.
A 30-chapter masterclass on building a large language model, and an engineering course library covering transformer internals, attention, quantization, fine-tuning and inference.
Get the exact numbers from the tools, and learn the underlying principles in the courses.
Planning multi-node clusters or modeling production workloads?
Recent analysis
olve ChatGPT action discovery probe errors by handling POST discovery probes with valid JSON-RPC fallback responses
The updated AI Engagement Index ranks nations by their engagement in technical AI content, offering a fresh perspective on the global AI adoption.
The fundamental algorithms of machine learning to build efficient, scalable, and highly optimized technical solutions.