The ApX VRAM Calculator models GPU memory allocation, generation throughput, prompt latency, and training step times before you provision or purchase hardware. Total GPU allocation is not bounded by quantized weight size alone: a serving process reserves a usable memory ceiling, retains 16-bit precision for embeddings and untied output heads below 256,000 vocabulary tokens, allocates driver contexts and CUDA graph capture pools, and partitions remaining VRAM into fixed-size KV pages.
Alongside single-rig sizing, the constraint-based setup planner searches GPU SKUs, node counts, quantizations, tensor parallelism, and pipeline parallelism degrees against target TTFT, per-user tok/s, VRAM headroom, and hourly budget. Candidate configurations that violate a physical or SLA bound are returned with explicit rejection reasons.
Each run outputs runtime diagnostic advisories for prefill starvation, PCIe all-reduce bottlenecks, FP8 software unpacking, KV pool preemption knees, and outer-stage pipeline asymmetry.
| Workload | Models covered | Primary sizing drivers | Reported metrics |
|---|---|---|---|
| Autoregressive Text & Multimodal Inference | Dense, MoE, State Space, Vision-Language, Audio-Language | Weight precision, KV cache precision, context length, batch size, concurrency, serving engine | Total and per-GPU VRAM, decode tok/s, TTFT, P50/P95/P99 TTFT, sustainable QPS |
| Non-Generative Single-Pass Inference | Embedding, Reranking, Decision models | Sequence length, batch size, weight precision | VRAM with zero KV cache allocation, pass latency, samples/s |
| Model Fine-Tuning | Full Fine-Tuning, LoRA, QLoRA | Optimizer type, ZeRO stage, micro-batch size, gradient accumulation, sequence length, LoRA rank | Peak VRAM, training tok/s, samples/s, step latency, Tensor Core MFU |
| Image & Video Diffusion | DiT and UNet backbones, inference and fine-tuning | Resolution, frame count, denoising steps, guidance scale, VAE tiling, encoder offload | Peak VRAM, total generation time, step latency, iterations/s, latent patch tokens |
| Platform class | Representative hardware | Memory & driver characteristics |
|---|---|---|
| NVIDIA Datacenter | H100, H200, B200, A100, L40S, L4 | Dedicated HBM or ECC GDDR6, per-GPU serving workers, NCCL collective support |
| NVIDIA Consumer | RTX 5090, 4090, 3090, workstation RTX Ada | Dedicated GDDR6X or GDDR7, PCIe-bound multi-GPU scaling |
| AMD Datacenter | Instinct MI300X, MI325X, MI350X, MI250 | High-capacity HBM3/HBM3e, ROCm / rocBLAS stack, xGMI fabric |
| AMD Consumer & APU | Radeon RX 7900 XTX, RX 9070, Radeon AI Pro, Ryzen AI | Discrete GDDR6 or unified LPDDR5X shared with system memory |
| Apple Silicon Unified Memory | M1 through M5 Pro, Max, and Ultra | Unified SoC RAM capped at 75% usable GPU memory by default, Metal / MPS |
When entering Custom Hardware specs, enter dense FP16 or BF16 Tensor Core TFLOPS rather than FP32 vector rates or 2:4 sparse marketing peaks.
The calculator separates static resident parameter memory from active per-token compute and dynamic attention state. Weight precision, unquantized projection rules, and attention geometry jointly determine both the minimum VRAM floor and the bandwidth required per generated token.
Mixture-of-Experts models hold all routed and shared experts in VRAM while streaming only active experts per token. As batch size rises, concurrent tokens activate more distinct experts per step, shifting MoE decode from active-expert bandwidth toward resident-weight bandwidth.
Under weight quantization, token embeddings, untied lm_head projections below the 256,000-token vocabulary threshold, and multimodal vision or audio encoders remain at 16-bit precision at <mBytes/>:
For custom or modified architectures, you can override layer count, hidden dimension, FFN intermediate size, vocabulary size, query heads, and KV heads directly.
At long contexts or high concurrency, KV cache memory frequently exceeds model weight memory. For Multi-Head, Grouped-Query, and Multi-Query Attention, per-token per-layer KV memory scales with KV head count <mNkv/>, head dimension <mDhead/>, and KV precision bytes <mBkv/>:
Multi-Head Latent Attention compresses key and value projections into a joint latent vector of dimension <mDc/> plus a decoupled rotary key of dimension <mDr/>, while Sliding Window Attention caps windowed layers at window length <mW/>:
DeepSeek Sparse Attention splits layers across windowed, 4:1 compressed, and 128:1 compressed tiers with sparse indexer keys <mDidx/>, while Compressed Sparse Attention 2 retains latent KV states only on <mLsrc/> source layers and stores indexer keys on downstream layers. Hybrid models scale KV growth by the full-attention layer fraction <mFfull/>:
Standard and grouped FP16 KV cache per user for a 64-layer, 8192-dim model
| Context | MHA | GQA 8:1 | MQA 64:1 |
|---|---|---|---|
| 4k | 8 GB | 1 GB | 0.13 GB |
| 16k | 32 GB | 4 GB | 0.5 GB |
| 32k | 64 GB | 8 GB | 1 GB |
| 64k | 128 GB | 16 GB | 2 GB |
| 128k | 256 GB | 32 GB | 4 GB |
Compressed and windowed FP16 KV cache per user for a 64-layer model
| Context | MLA | SWA 4k | DSA | CSA2 |
|---|---|---|---|---|
| 4k | 0.28 GB | 1 GB | 0.05 GB | 0.02 GB |
| 16k | 1.13 GB | 1 GB | 0.16 GB | 0.08 GB |
| 32k | 2.25 GB | 1 GB | 0.32 GB | 0.16 GB |
| 64k | 4.5 GB | 1 GB | 0.63 GB | 0.33 GB |
| 128k | 9 GB | 1 GB | 1.25 GB | 0.65 GB |
| Mechanism | Code | KV storage geometry | Capacity impact |
|---|---|---|---|
| Multi-Head Attention | MHA | Full K and V vector per query head | Highest KV growth; saturates VRAM at long context |
| Grouped-Query Attention | GQA | Shared K and V heads across query groups | Typically 4x to 8x smaller KV cache than MHA |
| Multi-Query Attention | MQA | Single K and V head across all queries | Minimal per-head KV footprint |
| Multi-Head Latent Attention | MLA | Joint low-rank compressed KV latent vector | Reduces KV cache by roughly 20x to 25x vs. MHA |
| DeepSeek Sparse Attention | DSA | Three-tier layer mix: sliding window, 4:1 compressed, and 128:1 compressed layers with sparse indexer keys | Sub-linear KV scaling at extreme context lengths |
| Compressed Sparse Attention 2 | CSA2 | Cross-layer latent KV source sharing paired with sparse indexer keys on downstream layers | Reuses source-layer KV states across layers for minimal multi-layer footprint |
| Sliding Window Attention | SWA | Caps windowed layers at sliding window length | Prevents unbounded KV growth past the window |
| Hybrid Linear / Full Attention | Hybrid | Mixes recurrent linear-attention layers with periodic full-attention layers | Scales KV cache down by the full-attention layer fraction |
Pure State Space Models such as Mamba SSM replace attention KV caches with fixed-size recurrent state buffers <mSsm/>, keeping memory constant regardless of sequence length.
Weight precision and KV cache precision are configured independently. Execution speed depends on whether the selected GPU has native Tensor or Matrix Core support for that format:
| Format | Bytes per param | Applies to | Silicon & execution behavior |
|---|---|---|---|
FP32 | 4.00 | Weights, master weights | Full-precision baseline |
FP16, BF16 | 2.00 | Weights, KV, activations | Standard 16-bit Tensor / Matrix Core rate |
FP8 | 1.00 | Weights, KV cache | Native 2x rate on Ada, Hopper, Blackwell, CDNA3, CDNA4, RDNA4; emulated elsewhere |
NVFP6 | 0.75 | Weights | 6-bit micro-scaled floating point |
NVFP4 | 0.50 | Weights | Native 4x Tensor Core rate on Blackwell and CDNA4 |
INT8, Q8 | 1.00 | Weights, KV cache | W8A16 unpacks to 16-bit registers before GEMM |
INT4 AWQ/GPTQ | 0.50 | Weights | W4A16 dequantizes to 16-bit on each step |
GGUF Q6_K to Q2_K | 0.8125 to 0.32 | GGUF weights | Block-quantized K-quants including quantized embeddings in llama.cpp |
Q4 KV cache | 0.50 | KV cache | 4-bit KV storage for maximum context concurrency |
FP8 weights to 16-bit at runtime and reduce generation speed.INT8 and INT4 reduce memory bandwidth at low batch sizes but incur per-step dequantization stalls, especially under eager PyTorch execution.RTX 30xx cards accumulate training gradients in FP32 at half their catalog FP16-accumulate peak.Every workload partitions usable device memory into static shared allocations and dynamic per-request or per-step pools. The calculator accounts for tensor weights, KV pages, optimizer states, activation peaks, and runtime driver reserves before evaluating fit.
Inference VRAM is split into Static Shared Memory and Per-User Dynamic Memory. When Prefix Caching is enabled, shared system prompt tokens <mShared/> are stored once across <mLayers/> layers, while unique per-user tokens <mUnique/> scale with active users <mUsers/>:
Prefix sharing aligns to 16-token or 32-token KV page boundaries depending on the runtime, so unaligned tail tokens remain unshared.
Physical Device VRAM splits via runtime cap into Usable Budget, which partitions into Static Shared Memory and Dynamic Request Pool.
| Memory bucket | Scope | Sizing behavior |
|---|---|---|
| Base / Expert Model Weights | Static Shared | Transformer layers, unquantized embeddings, and multimodal towers |
| Draft Model Weights | Static Shared | Resident speculative draft model when speculative decoding is enabled |
| KV Cache | Per-User Dynamic | Attention state per active stream, or full pre-allocated pool up to runtime limit |
| Activations | Dynamic | Bounded by chunk budget from 256 to 16,384 tokens under Chunked Prefill; scales with full prompt otherwise |
| Logits Buffer | Dynamic | Output vocabulary projection sized for final token or full prefill sequence |
| CUDA Graph Buckets | Static Shared | Captured graph pool reserved in Graph Captured mode across batch buckets |
Worked example: serving a 70B FP8 model on a single 80 GB H100 under vLLM appears to leave 10 GB free when multiplying 70B by 1 byte. In practice, vLLM caps usable VRAM at 92% or 73.6 GB, while unquantized 16-bit embeddings, lm_head weights, driver context, and CUDA graph capture leave minimal KV block headroom for concurrent 32k-token requests. Inspect this configuration directly in the VRAM Calculator.
FP32 master weights under mixed precision. With 32-bit AdamW across <mP/> parameters, static training state totals <m16p/> bytes before activations, dropping to <m6p/> bytes under pure BF16 or <m8p/> bytes with 8-bit optimizer states.r adapters <mPadapter/>, allocating gradients and optimizer states strictly to <mPadapter/>. QLoRA compresses the frozen transformer body to 4-bit and adds QLoRA Buffers for on-the-fly dequantization.FP32 vocabulary logits across the micro-batch.| Optimizer | State storage | Master weights | Sizing trade-off |
|---|---|---|---|
| AdamW | 32-bit momentum and variance | FP32 copy for mixed precision | Highest VRAM requirement; baseline convergence |
| 8-bit Optimizer | 8-bit quantized states | FP32 copy for mixed precision | Cuts momentum and variance memory by 4x |
| Paged Optimizer | 8-bit or 32-bit states | Host-paged under memory pressure | Absorbs activation spikes without crashing |
| Pure BF16 Training | 16-bit states | None | Eliminates FP32 master weights and halves state precision |
| SGD | 32-bit momentum | FP32 copy for mixed precision | Single state tensor instead of two |
| Adafactor | Factored row and column accumulators | FP32 copy for mixed precision | Sub-linear second-moment storage |
Worked example: serving a 70B FP8 model on a single 80 GB H100 under vLLM appears to leave 10 GB free when multiplying 70B by 1 byte. In practice, vLLM caps usable VRAM at 92% or 73.6 GB, while unquantized 16-bit embeddings, lm_head weights, driver context, and CUDA graph capture leave minimal KV block headroom for concurrent 32k-token requests. Inspect this configuration directly in the VRAM Calculator.
Diffusion memory tracks DiT Backbone Weights, 16-bit Text Encoder Weights, Latent Activations, and the VAE Decode Workspace. Patch attention memory scales with the 2D or 3D latent patch token count <mSpatches/> from width <mW/>, height <mH/>, and frames <mT/> using <mFs/>, <mPs/>, <mPt/>, and <mFt/> for video when <mTgt1/>:
Classifier-Free Guidance with guidance_scale > 1.0 doubles effective batch activations. Under sequential CPU offload, peak VRAM drops from the sum of all components to the maximum of the text encoder or DiT backbone plus active latent and VAE workspaces. VAE Tiling bounds volumetric decode memory on high-resolution images and video clips.
Every active GPU process reserves non-tensor memory before serving begins. Across multi-GPU nodes, these reserves multiply per process and consume 0.5 GB to several gigabytes per device.
| Sub-bucket | Vendor stack labels | Physical purpose |
|---|---|---|
| Driver Context | CUDA, ROCm, Metal Driver Context | Per-device page tables, command queues, virtual address space |
| BLAS Workspaces | cuBLAS, rocBLAS, MPS Workspaces | Pre-allocated GEMM scratchpad buffers |
| Allocator Retention | Allocator Retention, Unified Allocator Retention | Caching pool headroom preventing OS page thrashing |
| Operator Workspaces | Operator Workspaces | Temporary scratchpads for normalization, RoPE, and attention |
Worked example: serving a 70B FP8 model on a single 80 GB H100 under vLLM appears to leave 10 GB free when multiplying 70B by 1 byte. In practice, vLLM caps usable VRAM at 92% or 73.6 GB, while unquantized 16-bit embeddings, lm_head weights, driver context, and CUDA graph capture leave minimal KV block headroom for concurrent 32k-token requests. Inspect this configuration directly in the VRAM Calculator.
Serving engines, diffusion pipelines, and training frameworks are modeled through shared runtime traits: dispatch mode, KV paging policy, prefill chunk budget, usable memory ceiling, prefix caching, and process topology. A framework preset applies a named set of defaults across those traits, and any individual trait can be overridden field by field without leaving the preset.
Framework presets (vLLM, SGLang, TensorRT-LLM, Ollama, HF Transformers) map onto shared runtime traits: CUDA Graphs, Paged KV + Prefix Cache, Chunked Prefill, and Eager / Unpaged.
| Engine | Dispatch | Prefill behavior | KV paging | Usable memory policy | Prefix cache & process model |
|---|---|---|---|---|---|
| vLLM | CUDA Graphs | Chunked prefill at 2,048, 8,192, or 16,384 tokens by GPU VRAM tier | 16-token blocks | 92% of total VRAM | Per-GPU workers; block-hash prefix caching on |
| SGLang | CUDA Graphs | Chunked prefill | Paged pool | 88% static VRAM fraction | Per-GPU workers; RadixAttention tree prefix cache |
| TensorRT-LLM | Captured engine | Chunked prefill off by default; rejects prompts over 8,192 max_num_tokens | 32-token blocks | 90% of free VRAM after weights | Per-GPU workers; block reuse on |
| llama.cpp | CUDA Graphs | 512-token micro-batches | Unified context pool | Full available VRAM | Shared process; input tok_embd kept in host RAM |
| Ollama | CUDA Graphs | 512-token micro-batches | 4k under 23 GiB, 32k at 23+ GiB, 256k at 47+ GiB | Full available VRAM | Shared process; single-request default concurrency |
| HF Transformers | Eager PyTorch | Full prompt in one pass | Unpaged DynamicCache | Full available VRAM | Shared process with layer split; static batching |
| Generic | Hardware-inferred | Hardware-inferred | Hardware-inferred | Full available VRAM | Shared on consumer/Metal; per-GPU on datacenter |
When Runtime Preset is set to Auto, the calculator chooses llama.cpp for Apple Silicon or GGUF quantizations on single consumer GPUs. It selects Ollama for standard quantizations on single NVIDIA consumer GPUs, and vLLM for datacenter GPUs or multi-GPU NVIDIA consumer rigs.
| Engine | Execution strategy | Memory policy | Auto-selection rule |
|---|---|---|---|
| Diffusers Pipeline | Sequential pipeline with PyTorch SDPA | PyTorch caching allocator; sequential sub-module CPU offload | 24 GB to 47 GB consumer GPUs and fine-tuning |
| ComfyUI Node Engine | Unified node execution graph | Dynamic node-level weight swapping; zero cross-stage residual memory | Consumer GPUs under 24 GB VRAM |
| TensorRT Diffusion | Captured CUDA Graphs with static FMHA | 95% usable memory fraction; async multi-stream VAE decode | Datacenter GPUs or 48 GB+ cards |
Worked example: serving a 70B FP8 model on a single 80 GB H100 under vLLM appears to leave 10 GB free when multiplying 70B by 1 byte. In practice, vLLM caps usable VRAM at 92% or 73.6 GB, while unquantized 16-bit embeddings, lm_head weights, driver context, and CUDA graph capture leave minimal KV block headroom for concurrent 32k-token requests. Inspect this configuration directly in the VRAM Calculator.
| Framework | Attention & loss | Checkpointing & packing | Default optimizer & precision | Auto-selection rule |
|---|---|---|---|---|
| HF Transformers + PEFT | PyTorch SDPA; full FP32 loss logits | Checkpointing off; dynamic padding | 32-bit AdamW; 16-bit embeddings under QLoRA | Single-GPU consumer Full Fine-Tuning |
| Unsloth | Fused Triton kernels; chunked cross-entropy | Host-offloaded checkpointing; sequence packing on | Paged 8-bit optimizer | Single-GPU LoRA and QLoRA |
| torchtune | Flash / SDPA; linear cross-entropy loss | Activation checkpointing on; packing off | Pure BF16 weights and optimizer states | Datacenter Full Fine-Tuning and multi-GPU/multi-node training |
Across <mNdp/> data-parallel GPUs, DeepSpeed ZeRO Stage 1 divides optimizer states by <mNdp/>. ZeRO Stage 2 divides optimizer states and gradients by <mNdp/>, and ZeRO Stage 3 or PyTorch FSDP divides parameters, gradients, optimizer states, and checkpointed activation boundaries by <mNdp/>.
| Control | Options | Practical effect |
|---|---|---|
| Execution Mode | Graph Captured vs. Dynamic Eager | Graph capture eliminates Python/CPU dispatch overhead at the cost of static CUDA graph pool VRAM |
| Prefill Mode | Chunked 256 to 16,384 tokens, or Full Sequence | Chunking caps prefill activation spikes and prevents long prompts from stalling active decode streams |
| GPU Memory Reserved | Fraction from 0.50 to 0.98 | Sets how much VRAM the engine reserves for weights plus KV block pools before rejecting new requests |
| Prefix Caching | On or Off | Reuses KV blocks for shared system prompts, reducing per-user KV growth and TTFT |
| Continuous Batching | Continuous vs. Static | Iteration-level scheduling improves aggregate throughput under uneven request lengths |
| Static VRAM Pre-allocation | On-Demand vs. Pre-allocated fraction | Models engines that claim the full KV memory pool upfront at startup |
Throughput and latency follow physical hardware rooflines coupled with runtime scheduling behavior. The performance model covers single-stream decode speed, prompt prefill latency, and tail-latency cliffs as concurrent KV pools approach saturation.
Autoregressive decode throughput is bounded by hardware memory bandwidth <mBw/> and dense compute <mFlops/> scaled by calibrated silicon, quantization, and batch efficiency terms <mEtaBw/> and <mEtaCompute/>:
Normalized decode throughput vs. batch size as execution shifts from memory-bandwidth bound to compute bound
| KV pool utilization | P50 TTFT multiplier | P99 TTFT multiplier |
|---|---|---|
| 20% | 1x | 1.2x |
| 40% | 1.1x | 1.5x |
| 60% | 1.3x | 2.1x |
| 80% | 1.7x | 3.2x |
| 95% | 2.4x | 5.1x |
| 100% | 3.1x | 7.4x |
| 110% | 6.8x | 18.5x |
The transition from memory-bandwidth bound to compute-bound decode occurs at the arithmetic-intensity crossover batch size <mBstar/>:
For Mixture-of-Experts models, rising batch size <mB/> increases the count of distinct routed experts activated per step, shifting decode bandwidth demand smoothly from active-expert weights toward the full resident expert pool.
Prompt prefill is compute-bound across linear projections and quadratic <mSseq2/> self-attention at long contexts:
For Diffusion Transformers, each denoising step combines linear backbone FLOPs <mDitLinear/> with quadratic patch attention <mDitAttn/>.
Effective bandwidth, compute utilization, and dispatch overheads are calibrated against physical hardware benchmarks on the Accuracy Scorecard.
Remaining usable VRAM is divided into fixed-size KV blocks such as 16 tokens in vLLM or 32 tokens in TensorRT-LLM, with each user's unique non-shared tokens rounded up to whole blocks. As request load approaches theoretical service capacity, continuous-batching queue delay rises sharply and drives a tail-latency cliff at the Safe Operating Knee; sustainable QPS is bounded below that saturation knee.
Relative TTFT latency multiplier vs. KV block pool utilization across the safe operating knee
| KV pool utilization | P50 TTFT multiplier | P99 TTFT multiplier |
|---|---|---|
| 20% | 1x | 1.2x |
| 40% | 1.1x | 1.5x |
| 60% | 1.3x | 2.1x |
| 80% | 1.7x | 3.2x |
| 95% | 2.4x | 5.1x |
| 100% | 3.1x | 7.4x |
| 110% | 6.8x | 18.5x |
The concurrency analyzer tracks allocator admission occupancy watermarks for moderate and critical preemption risk, reports P50, P95, and P99 TTFT, and flags prefill starvation when unchunked long prompts stall active decode streams.
Sequential drafting uses a smaller standalone model to propose <mGamma/> tokens verified in parallel by the target model. Block-parallel drafting such as DFlash, Medusa, and Eagle proposes a token block in a single pass.
Both methods raise low-batch decode speed, but resident draft weights shrink the free KV block pool and lower the preemption-free concurrency ceiling.
Training step latency splits into four phases: Forward Pass, Backward Pass including gradient-checkpointing recomputation, Collective Comms across TP/PP/DP groups, and Optimizer Step including host offload transfers. ZeRO Stage 3 and FSDP model both All-Gather parameter and Reduce-Scatter gradient traffic across <mNdp/> devices, whereas ZeRO Stage 2 models Reduce-Scatter gradient traffic alone, flagging steps that become communication-bound.
Sweeping micro-batch sizes from 1 to 32 against remaining activation headroom plots Tensor Core MFU saturation, identifies the largest safe micro-batch before backward activations OOM, and computes the gradient accumulation steps required to reach your target global batch size.
Multi-device setups span Single Rig workstations and multi-node Clusters partitioned across Tensor Parallelism, Pipeline Parallelism, Sequence Parallelism, and Data Parallelism.
| Strategy | What is split | Communication pattern | When to use |
|---|---|---|---|
| Tensor Parallelism | Attention heads, MLP matrices, KV cache | All-reduce on every layer | Lowest single-user latency inside an NVLink or xGMI node |
| Pipeline Parallelism | Layer groups across stages | Point-to-point stage handoff | Spanning PCIe links or multiple nodes |
| Sequence Parallelism | 2D and 3D latent patch tokens in DiT | Ring or Ulysses attention exchange | High-resolution image and video diffusion |
| Data Parallelism | Independent replicas or ZeRO shards | None in inference; ZeRO collectives in training | Scaling aggregate QPS or training batch size |
Under Pipeline Parallelism across <mNpp/> stages, Stage 0 holds input embeddings and the Final Stage holds lm_head on top of their layer share. Outer stages OOM first if a cluster is sized by average per-GPU memory:
Per-stage weight memory for a 70B FP16 model with 128k untied vocabulary across 4 pipeline stages
| Stage | Transformer layers | Embeddings / lm_head |
|---|---|---|
| Stage 0 | 35 GB | 2 GB |
| Stage 1 | 35 GB | 0 GB |
| Stage 2 | 35 GB | 0 GB |
| Stage 3 | 35 GB | 2 GB |
In a 4-stage pipeline on a 70B FP16 model with a 128k untied vocabulary, middle stages hold 35.0 GB of layers while Stage 0 and Stage 3 hold 37.0 GB each. Sizing purely by the 35.5 GB average underestimates outer-stage peak demand by 1.5 GB per GPU. Evaluate multi-GPU splits in the VRAM Calculator.
| Tier | Supported interconnects | Viable parallelism |
|---|---|---|
| High-Speed Fabrics | NVLink Gen 2 through Gen 6, NVLink Bridge, AMD Infinity Fabric xGMI, InfiniBand EDR through XDR | Tensor Parallelism, Sequence Parallelism, FSDP / ZeRO-3 |
| PCIe | PCIe 3.0, 4.0, and 5.0 x16 | Pipeline Parallelism, Data Parallelism, low-degree TP |
| Network & External Links | Thunderbolt 4 and 5, Ethernet 1G through 400G | Pipeline Parallelism or replicated Data Parallelism |
When Tensor Parallelism, Pipeline Parallelism stages, or FSDP/ZeRO replicas span multiple physical nodes, communication efficiency is blended between the intra-node fabric and inter-node network according to the exact fraction of cross-node hops.
Multi-node configurations also output a Disaggregated Prefill/Decode Network Budget detailing per-request KV transfer volume, required fabric bandwidth, and minimum NIC line rate from 100G Ethernet to 800G InfiniBand / RoCE.
Node 0 and Node 1 run intra-node TP and PP over NVLink or xGMI, while cross-node PP, DP/ZeRO, and KV transfers traverse InfiniBand or Ethernet.
| Offload target | Granularity | Performance impact |
|---|---|---|
| Model Weights | Layer count or percentage to System RAM / NVMe; prioritizes MoE expert weights before shared layers | Shifts streamed layer reads from HBM to PCIe / NVMe |
| KV Cache | Full KV pool to System RAM | Transfers attention state over PCIe on every token, severely reducing decode tok/s |
| ZeRO Training States | Optimizer states, gradients, and activations to CPU / NVMe | Adds PCIe transfer latency to the optimizer and backward steps |
Worked example: serving a 70B FP8 model on a single 80 GB H100 under vLLM appears to leave 10 GB free when multiplying 70B by 1 byte. In practice, vLLM caps usable VRAM at 92% or 73.6 GB, while unquantized 16-bit embeddings, lm_head weights, driver context, and CUDA graph capture leave minimal KV block headroom for concurrent 32k-token requests. Inspect this configuration directly in the VRAM Calculator.
Predictions are continuously verified against physical hardware traces, with live test cases and error distributions published on the Accuracy Scorecard.
| Suite | What is verified | Ground-truth source |
|---|---|---|
| Suite A: Model Weights | Static weight footprints | Published .safetensors and GGUF artifact sizes |
| Suite B: KV Cache | Attention KV scaling | Exact tensor geometry across MHA, GQA, MQA, and MLA |
| Suite C / N: Decode Speed | Inference throughput and TTFT | Physical runs across Apple Silicon, consumer GPUs, and datacenter GPUs |
| Suite D: Fine-Tuning VRAM | Training peak memory | Profiler traces across ZeRO-3, LoRA, QLoRA, and Full Fine-Tuning |
| Suite E: End-to-End Inference VRAM | Total serving allocation | torch.cuda.max_memory_allocated and engine memory telemetry |
| Suite T: Training Speed | Step latency and tok/s | Measured runs across torchtune, HF FSDP, DeepSpeed, and NeMo |
Each suite is scored by In-Band Pass Rate, Median Error, Geometric Mean Fold Error, and Systemic Bias. Representative ground-truth checks from the active suite:
| Case ID | Configuration | Measured | Predicted |
|---|---|---|---|
A1 | Llama-3-8B FP16 safetensors weights | 14.97 GB | 14.96 GB |
A2 | Qwen2.5-32B Q4_K_M GGUF weights | 18.49 GB | 18.18 GB |
E2 | Qwen2.5-32B Q4_K_M end-to-end 16k ctx | 24.00 GB | 23.20 GB |
C3 | LLaMA-7B Q4_0 on M2 Ultra 76c, llama.cpp | 94.3 tok/s | 96.0 tok/s |
D1 | Llama-3.1-405B FP8, 8x H200 SXM, TensorRT-LLM TP=8 | 56.0 tok/s | 55.0 tok/s |
Memory capacity predictions achieve sub-1% median error because tensor dimensions, KV block pages, optimizer states, and driver contexts follow deterministic physical bounds.
Throughput and training speed estimates carry wider variance, typically 14% to 24%, due to host CPU dispatch jitter, thermal and electrical throttling, background PCIe contention, and undocumented runtime flags.
For programmatic integration into CI/CD pipelines or autonomous agents, see the REST API Documentation and MCP Server Documentation.
Last updated: November 17, 2026