Precision for model weights during inference. Lower uses less VRAM but may affect quality.
KV Cache precision. Lower values reduce VRAM, especially for long sequences.
Hardware Configuration
Select your GPU or set custom VRAM
Num GPUs
Devices for parallel inference
Input Parameters
Batch Size:
Inputs processed simultaneously per step
Sequence Length:
Context window size. Includes input and output tokens.
Concurrent Users:
Number of users running inference simultaneously
▶ Advanced Configuration
(FP16 Weights / FP16 KV Cache) on 16GB / 360 GB/s / 26 TFLOPS Custom GPU
Input sequence length: 1,024 tokens
Configure model and hardware to enable simulationThe ApX Calculator models GPU memory consumption, distributed cluster topologies, and inference latency using analytical formulas validated against empirical PyTorch, vLLM, and TensorRT-LLM execution traces.
Static resident VRAM depends on total parameter count, precision bit-width, and quantization tensor scale overhead.
VRAM_weights = (Parameters * Bits_per_Weight) / (8 * 1024³) * (1 + Overhead)Precision formats supported include FP16/BF16 (16-bit), FP8 E4M3/E5M2 (8-bit), INT8 (8-bit), INT4/AWQ/GPTQ (4-bit + scale tensors), and GGUF quantization levels (Q2_K through Q8_0).
KV cache scales with sequence length, concurrent batch size, and attention topology.
// MHA / GQA:
KV_Cache = 2 * Layers * KV_Heads * Head_Dim * Batch * Seq_Len * Bytes
// MLA (DeepSeek-V2 / V3):
KV_Cache = Layers * (d_c + d_R) * Batch * Seq_Len * BytesAccounts for Multi-Head Attention (MHA), Grouped-Query Attention (GQA), and Multi-Head Latent Attention (MLA) with latent KV compression.
Multi-GPU cluster configurations distribute memory and compute across nodes:
| Strategy | Memory Distribution | Primary Interconnect |
|---|---|---|
| Tensor Parallelism (TP) | Splits weights and KV cache by N_TP | NVLink / Intra-node |
| Pipeline Parallelism (PP) | Partitions layers across nodes | PCIe / InfiniBand |
| Expert Parallelism (EP) | Distributes MoE routed experts | All-to-All Fabric |
| Context Parallelism (CP) | Segments long-context token sequences | Ring Attention / Collective |
Inference execution separates into compute-bound prefill and bandwidth-bound decode:
Prefill_Latency ≈ (2 * Params * Prompt_Tokens) / Effective_TFLOPS
Decode_Throughput ≈ GPU_Bandwidth_GBs / (Weights_GB + Active_KV_GB)Simulates Time-to-First-Token (TTFT), tokens per second per stream, speculative decoding acceptance rates, and offloading latency across PCIe and NVMe buses.
Most basic tools only perform naive multiplication of parameter counts and bit-widths. The ApX VRAM & Infrastructure Calculator models full dynamic runtime allocations, including attention topology variations (MHA, GQA, and DeepSeek MLA), activation checkpoints, distributed multi-node parallelism splits (TP, PP, DP, EP, CP), quantization metadata scaling overhead, speculative decoding speedups, and real-world benchmarked execution constraints verified against vLLM and PyTorch.
Unlike standard Multi-Head Attention (MHA) which stores separate key and value vectors for every head, MLA projects keys and values into a compressed latent space of dimension d_c (e.g. 512) and appends a decoupled rotary key vector d_R (e.g. 64). During generation, only (d_c + d_R) elements are retained in the KV cache per token per layer, reducing dynamic memory consumption by 80% to 90% compared to MHA.
Tensor Parallelism shards linear layer matrix multiplications across GPUs. Each forward pass requires two All-Reduce communications per transformer layer. Because of the frequent collective communication, Tensor Parallelism is typically constrained to GPUs connected via high-bandwidth intra-node links such as NVLink (900 GB/s on Hopper H100, 450 GB/s on Ampere A100).
Full parameter fine-tuning with 32-bit AdamW requires approximately 16 to 18 bytes per parameter (2 bytes weights, 2 bytes gradients, 12 bytes optimizer states, plus activation memory). LoRA freezes base weights at 16-bit and allocates optimizer states only for rank-r adapter matrices. QLoRA further quantizes the base model to 4-bit NormalFloat (NF4), reducing base parameter footprint to ~0.55 bytes per parameter while maintaining 16-bit or 32-bit adapter training.
During single-token decoding, every layer weight must be transferred from GPU VRAM to the compute registers to perform matrix-vector arithmetic with an arithmetic intensity close to 1 FLOP/byte. The maximum theoretical token generation speed is governed by GPU memory bandwidth divided by total active memory transferred per step.
For MoE architectures, all expert weights remain resident in memory across GPUs (or partitioned with Expert Parallelism EP). Dynamic computation FLOPs and active KV cache are calculated strictly based on the top-k activated experts per token, incorporating all-to-all dispatch overhead and shared attention layers.
Apple Silicon Macs (e.g. M2/M3/M4 Max and Ultra with up to 128 GB or 192 GB) share system memory directly between CPU and GPU without PCIe bottlenecking. The calculator models macOS memory wiring limits, allowing sizing for 70B models at INT4/Q4_K_M (~40 GB footprint) to verify unified memory and bandwidth headroom.
The integrated Accuracy & Perplexity Scorecard tracks exact memory savings alongside measured perplexity degradation on standard benchmarks (such as Wikitext-2 and C4), empirically evaluating the trade-off between VRAM reduction and output fidelity.
Using roofline capacity modeling, the calculator divides the memory streamed per token by aggregate cluster memory bandwidth across Tensor Parallelism (TP) shards. The setup planner suggests optimal cluster dimensions or quantization formats if bandwidth limits fall short of latency SLAs.
GPU specifications often list either traditional FP32 compute performance or sparse FP16 Tensor Core performance. Traditional FP32 performance is too slow to model LLM execution, which runs in half-precision or lower. Sparse FP16 specs assume hardware-level 2:4 sparsity, which standard LLM runners do not utilize by default. This calculator uses peak dense FP16/BF16 Tensor Core, AMD Matrix Core, or Apple Silicon native FP16 TFLOPS to ensure accurate speed predictions.
©2025 ApX Machine Learning
0.0%
VRAM
0 GB
of 12 GB VRAM
Generation Speed: ...
Time to First Token: ~0ms
Est. GPU Rental: N/A