LLM Inference: VRAM & Performance Calculator
Precision for model weights during inference. Lower uses less VRAM but may affect quality.
KV Cache precision. Lower values reduce VRAM, especially for long sequences.
Select your GPU or set custom VRAM
Devices for parallel inference
Batch Size
1
Inputs processed simultaneously per step (affects throughput & latency)
Sequence Length: 2,048
Max tokens per input; impacts KV cache (also affected by attention structure) & activations.
Concurrent Users
Number of users running inference simultaneously (affects memory usage and per-user performance)
0.0%
VRAM
0 GB
of 0 GB VRAM
Generation Speed: ...
Total Throughput: ...
Mode: Inference | Batch: 1
Inference Simulation
(FP16 Weights / FP16 KV Cache) on RTX 3060 (12GB)
Input sequence length: 2,048 tokens
Configure model and hardware to enable simulationHow Calculations Are Made
Memory usage is estimated using models that factor in architecture (parameters, layers, hidden dimensions, active experts, etc.), quantization, sequence length, and batch size. Performance estimates consider model/hardware analysis and benchmarks, though benchmark accuracy varies. Results are approximate.
Learn more about how VRAM requirements are calculated →Recent Updates
- 10 May 2025 - Fix DeepSeek V3 MOE calculation bug. Improve MOE calculation and enable precision selection for fine-tuning.
- 6 May 2025 - Add KV cache quantization customization, fix calculation for non-MHA attention structures.
- 3 May 2025 - Fix MOE calculation bug for Qwen 3. Add more GPU options.
- 2 May 2025 - Fix RTX 5090 VRAM. Temporarily disabled hidden layer size in calculation for Qwen 3*
* Newly released models may have more inaccuracies. Will improve the calculation for Qwen 3 once the model code is open sourced, to better understand the architecture.