VRAM Calculator Documentation

The ApX VRAM Calculator models GPU memory allocation, generation throughput, prompt latency, and training step times before you provision or purchase hardware. Total GPU allocation is not bounded by quantized weight size alone: a serving process reserves a usable memory ceiling, retains 16-bit precision for embeddings and untied output heads below 256,000 vocabulary tokens, allocates driver contexts and CUDA graph capture pools, and partitions remaining VRAM into fixed-size KV pages.

Alongside single-rig sizing, the constraint-based setup planner searches GPU SKUs, node counts, quantizations, tensor parallelism, and pipeline parallelism degrees against target TTFT, per-user tok/s, VRAM headroom, and hourly budget. Candidate configurations that violate a physical or SLA bound are returned with explicit rejection reasons.

Each run outputs runtime diagnostic advisories for prefill starvation, PCIe all-reduce bottlenecks, FP8 software unpacking, KV pool preemption knees, and outer-stage pipeline asymmetry.

Supported workloads

WorkloadModels coveredPrimary sizing driversReported metrics
Autoregressive Text & Multimodal InferenceDense, MoE, State Space, Vision-Language, Audio-LanguageWeight precision, KV cache precision, context length, batch size, concurrency, serving engineTotal and per-GPU VRAM, decode tok/s, TTFT, P50/P95/P99 TTFT, sustainable QPS
Non-Generative Single-Pass InferenceEmbedding, Reranking, Decision modelsSequence length, batch size, weight precisionVRAM with zero KV cache allocation, pass latency, samples/s
Model Fine-TuningFull Fine-Tuning, LoRA, QLoRAOptimizer type, ZeRO stage, micro-batch size, gradient accumulation, sequence length, LoRA rankPeak VRAM, training tok/s, samples/s, step latency, Tensor Core MFU
Image & Video DiffusionDiT and UNet backbones, inference and fine-tuningResolution, frame count, denoising steps, guidance scale, VAE tiling, encoder offloadPeak VRAM, total generation time, step latency, iterations/s, latent patch tokens

Hardware & platform classes

Platform classRepresentative hardwareMemory & driver characteristics
NVIDIA DatacenterH100, H200, B200, A100, L40S, L4Dedicated HBM or ECC GDDR6, per-GPU serving workers, NCCL collective support
NVIDIA ConsumerRTX 5090, 4090, 3090, workstation RTX AdaDedicated GDDR6X or GDDR7, PCIe-bound multi-GPU scaling
AMD DatacenterInstinct MI300X, MI325X, MI350X, MI250High-capacity HBM3/HBM3e, ROCm / rocBLAS stack, xGMI fabric
AMD Consumer & APURadeon RX 7900 XTX, RX 9070, Radeon AI Pro, Ryzen AIDiscrete GDDR6 or unified LPDDR5X shared with system memory
Apple Silicon Unified MemoryM1 through M5 Pro, Max, and UltraUnified SoC RAM capped at 75% usable GPU memory by default, Metal / MPS

When entering Custom Hardware specs, enter dense FP16 or BF16 Tensor Core TFLOPS rather than FP32 vector rates or 2:4 sparse marketing peaks.

Model architectures & quantization

The calculator separates static resident parameter memory from active per-token compute and dynamic attention state. Weight precision, unquantized projection rules, and attention geometry jointly determine both the minimum VRAM floor and the bandwidth required per generated token.

Dense transformers, Mixture-of-Experts, and multimodal towers

Mixture-of-Experts models hold all routed and shared experts in VRAM while streaming only active experts per token. As batch size rises, concurrent tokens activate more distinct experts per step, shifting MoE decode from active-expert bandwidth toward resident-weight bandwidth.

Under weight quantization, token embeddings, untied lm_head projections below the 256,000-token vocabulary threshold, and multimodal vision or audio encoders remain at 16-bit precision at <mBytes/>:

For custom or modified architectures, you can override layer count, hidden dimension, FFN intermediate size, vocabulary size, query heads, and KV heads directly.

Attention structures & hybrid state-space layers

At long contexts or high concurrency, KV cache memory frequently exceeds model weight memory. For Multi-Head, Grouped-Query, and Multi-Query Attention, per-token per-layer KV memory scales with KV head count <mNkv/>, head dimension <mDhead/>, and KV precision bytes <mBkv/>:

Multi-Head Latent Attention compresses key and value projections into a joint latent vector of dimension <mDc/> plus a decoupled rotary key of dimension <mDr/>, while Sliding Window Attention caps windowed layers at window length <mW/>:

DeepSeek Sparse Attention splits layers across windowed, 4:1 compressed, and 128:1 compressed tiers with sparse indexer keys <mDidx/>, while Compressed Sparse Attention 2 retains latent KV states only on <mLsrc/> source layers and stores indexer keys on downstream layers. Hybrid models scale KV growth by the full-attention layer fraction <mFfull/>:

Standard and grouped FP16 KV cache per user for a 64-layer, 8192-dim model

ContextMHAGQA 8:1MQA 64:1
4k8 GB1 GB0.13 GB
16k32 GB4 GB0.5 GB
32k64 GB8 GB1 GB
64k128 GB16 GB2 GB
128k256 GB32 GB4 GB

Compressed and windowed FP16 KV cache per user for a 64-layer model

ContextMLASWA 4kDSACSA2
4k0.28 GB1 GB0.05 GB0.02 GB
16k1.13 GB1 GB0.16 GB0.08 GB
32k2.25 GB1 GB0.32 GB0.16 GB
64k4.5 GB1 GB0.63 GB0.33 GB
128k9 GB1 GB1.25 GB0.65 GB
MechanismCodeKV storage geometryCapacity impact
Multi-Head AttentionMHAFull K and V vector per query headHighest KV growth; saturates VRAM at long context
Grouped-Query AttentionGQAShared K and V heads across query groupsTypically 4x to 8x smaller KV cache than MHA
Multi-Query AttentionMQASingle K and V head across all queriesMinimal per-head KV footprint
Multi-Head Latent AttentionMLAJoint low-rank compressed KV latent vectorReduces KV cache by roughly 20x to 25x vs. MHA
DeepSeek Sparse AttentionDSAThree-tier layer mix: sliding window, 4:1 compressed, and 128:1 compressed layers with sparse indexer keysSub-linear KV scaling at extreme context lengths
Compressed Sparse Attention 2CSA2Cross-layer latent KV source sharing paired with sparse indexer keys on downstream layersReuses source-layer KV states across layers for minimal multi-layer footprint
Sliding Window AttentionSWACaps windowed layers at sliding window lengthPrevents unbounded KV growth past the window
Hybrid Linear / Full AttentionHybridMixes recurrent linear-attention layers with periodic full-attention layersScales KV cache down by the full-attention layer fraction

Pure State Space Models such as Mamba SSM replace attention KV caches with fixed-size recurrent state buffers <mSsm/>, keeping memory constant regardless of sequence length.

Weight & KV cache quantization formats

Weight precision and KV cache precision are configured independently. Execution speed depends on whether the selected GPU has native Tensor or Matrix Core support for that format:

FormatBytes per paramApplies toSilicon & execution behavior
FP324.00Weights, master weightsFull-precision baseline
FP16, BF162.00Weights, KV, activationsStandard 16-bit Tensor / Matrix Core rate
FP81.00Weights, KV cacheNative 2x rate on Ada, Hopper, Blackwell, CDNA3, CDNA4, RDNA4; emulated elsewhere
NVFP60.75Weights6-bit micro-scaled floating point
NVFP40.50WeightsNative 4x Tensor Core rate on Blackwell and CDNA4
INT8, Q81.00Weights, KV cacheW8A16 unpacks to 16-bit registers before GEMM
INT4 AWQ/GPTQ0.50WeightsW4A16 dequantizes to 16-bit on each step
GGUF Q6_K to Q2_K0.8125 to 0.32GGUF weightsBlock-quantized K-quants including quantized embeddings in llama.cpp
Q4 KV cache0.50KV cache4-bit KV storage for maximum context concurrency
  • FP8 emulation on older silicon: GPUs without native 8-bit floating-point Tensor Cores, such as NVIDIA Ampere and Apple Silicon, unpack FP8 weights to 16-bit at runtime and reduce generation speed.
  • Weight-only INT8 and INT4 overhead: INT8 and INT4 reduce memory bandwidth at low batch sizes but incur per-step dequantization stalls, especially under eager PyTorch execution.
  • Architecture-specific kernels: Hopper GPUs use FlashAttention-3 paths for paged attention, while GeForce Ampere RTX 30xx cards accumulate training gradients in FP32 at half their catalog FP16-accumulate peak.

Memory allocation & overhead buckets

Every workload partitions usable device memory into static shared allocations and dynamic per-request or per-step pools. The calculator accounts for tensor weights, KV pages, optimizer states, activation peaks, and runtime driver reserves before evaluating fit.

Inference weights, paged KV cache, activations, and CUDA graphs

Inference VRAM is split into Static Shared Memory and Per-User Dynamic Memory. When Prefix Caching is enabled, shared system prompt tokens <mShared/> are stored once across <mLayers/> layers, while unique per-user tokens <mUnique/> scale with active users <mUsers/>:

Prefix sharing aligns to 16-token or 32-token KV page boundaries depending on the runtime, so unaligned tail tokens remain unshared.

Physical VRAM partitioning into usable pool, static shared footprint, and dynamic request allocation

Physical Device VRAM splits via runtime cap into Usable Budget, which partitions into Static Shared Memory and Dynamic Request Pool.

Memory bucketScopeSizing behavior
Base / Expert Model WeightsStatic SharedTransformer layers, unquantized embeddings, and multimodal towers
Draft Model WeightsStatic SharedResident speculative draft model when speculative decoding is enabled
KV CachePer-User DynamicAttention state per active stream, or full pre-allocated pool up to runtime limit
ActivationsDynamicBounded by chunk budget from 256 to 16,384 tokens under Chunked Prefill; scales with full prompt otherwise
Logits BufferDynamicOutput vocabulary projection sized for final token or full prefill sequence
CUDA Graph BucketsStatic SharedCaptured graph pool reserved in Graph Captured mode across batch buckets

Worked example: serving a 70B FP8 model on a single 80 GB H100 under vLLM appears to leave 10 GB free when multiplying 70B by 1 byte. In practice, vLLM caps usable VRAM at 92% or 73.6 GB, while unquantized 16-bit embeddings, lm_head weights, driver context, and CUDA graph capture leave minimal KV block headroom for concurrent 32k-token requests. Inspect this configuration directly in the VRAM Calculator.

Optimizer states, gradients, adapters, and training activations

  • Full Fine-Tuning: Stores model weights, full gradients, optimizer states, and FP32 master weights under mixed precision. With 32-bit AdamW across <mP/> parameters, static training state totals <m16p/> bytes before activations, dropping to <m6p/> bytes under pure BF16 or <m8p/> bytes with 8-bit optimizer states.
  • LoRA and QLoRA: LoRA freezes 16-bit base weights and trains rank-r adapters <mPadapter/>, allocating gradients and optimizer states strictly to <mPadapter/>. QLoRA compresses the frozen transformer body to 4-bit and adds QLoRA Buffers for on-the-fly dequantization.
  • Gradient Checkpointing and Loss Kernels: Gradient Checkpointing recomputes activations layer by layer during backpropagation to save VRAM. Chunked Cross-Entropy Loss avoids materializing full FP32 vocabulary logits across the micro-batch.
OptimizerState storageMaster weightsSizing trade-off
AdamW32-bit momentum and varianceFP32 copy for mixed precisionHighest VRAM requirement; baseline convergence
8-bit Optimizer8-bit quantized statesFP32 copy for mixed precisionCuts momentum and variance memory by 4x
Paged Optimizer8-bit or 32-bit statesHost-paged under memory pressureAbsorbs activation spikes without crashing
Pure BF16 Training16-bit statesNoneEliminates FP32 master weights and halves state precision
SGD32-bit momentumFP32 copy for mixed precisionSingle state tensor instead of two
AdafactorFactored row and column accumulatorsFP32 copy for mixed precisionSub-linear second-moment storage

Worked example: serving a 70B FP8 model on a single 80 GB H100 under vLLM appears to leave 10 GB free when multiplying 70B by 1 byte. In practice, vLLM caps usable VRAM at 92% or 73.6 GB, while unquantized 16-bit embeddings, lm_head weights, driver context, and CUDA graph capture leave minimal KV block headroom for concurrent 32k-token requests. Inspect this configuration directly in the VRAM Calculator.

DiT backbone weights, text encoders, latent patches, and VAE workspace

Diffusion memory tracks DiT Backbone Weights, 16-bit Text Encoder Weights, Latent Activations, and the VAE Decode Workspace. Patch attention memory scales with the 2D or 3D latent patch token count <mSpatches/> from width <mW/>, height <mH/>, and frames <mT/> using <mFs/>, <mPs/>, <mPt/>, and <mFt/> for video when <mTgt1/>:

Classifier-Free Guidance with guidance_scale > 1.0 doubles effective batch activations. Under sequential CPU offload, peak VRAM drops from the sum of all components to the maximum of the text encoder or DiT backbone plus active latent and VAE workspaces. VAE Tiling bounds volumetric decode memory on high-resolution images and video clips.

Granular driver context, BLAS workspaces, and allocator retention

Every active GPU process reserves non-tensor memory before serving begins. Across multi-GPU nodes, these reserves multiply per process and consume 0.5 GB to several gigabytes per device.

Sub-bucketVendor stack labelsPhysical purpose
Driver ContextCUDA, ROCm, Metal Driver ContextPer-device page tables, command queues, virtual address space
BLAS WorkspacescuBLAS, rocBLAS, MPS WorkspacesPre-allocated GEMM scratchpad buffers
Allocator RetentionAllocator Retention, Unified Allocator RetentionCaching pool headroom preventing OS page thrashing
Operator WorkspacesOperator WorkspacesTemporary scratchpads for normalization, RoPE, and attention

Worked example: serving a 70B FP8 model on a single 80 GB H100 under vLLM appears to leave 10 GB free when multiplying 70B by 1 byte. In practice, vLLM caps usable VRAM at 92% or 73.6 GB, while unquantized 16-bit embeddings, lm_head weights, driver context, and CUDA graph capture leave minimal KV block headroom for concurrent 32k-token requests. Inspect this configuration directly in the VRAM Calculator.

Runtime traits & framework presets

Serving engines, diffusion pipelines, and training frameworks are modeled through shared runtime traits: dispatch mode, KV paging policy, prefill chunk budget, usable memory ceiling, prefix caching, and process topology. A framework preset applies a named set of defaults across those traits, and any individual trait can be overridden field by field without leaving the preset.

Framework presets mapping onto shared orthogonal runtime traits

Framework presets (vLLM, SGLang, TensorRT-LLM, Ollama, HF Transformers) map onto shared runtime traits: CUDA Graphs, Paged KV + Prefix Cache, Chunked Prefill, and Eager / Unpaged.

Inference serving engines

EngineDispatchPrefill behaviorKV pagingUsable memory policyPrefix cache & process model
vLLMCUDA GraphsChunked prefill at 2,048, 8,192, or 16,384 tokens by GPU VRAM tier16-token blocks92% of total VRAMPer-GPU workers; block-hash prefix caching on
SGLangCUDA GraphsChunked prefillPaged pool88% static VRAM fractionPer-GPU workers; RadixAttention tree prefix cache
TensorRT-LLMCaptured engineChunked prefill off by default; rejects prompts over 8,192 max_num_tokens32-token blocks90% of free VRAM after weightsPer-GPU workers; block reuse on
llama.cppCUDA Graphs512-token micro-batchesUnified context poolFull available VRAMShared process; input tok_embd kept in host RAM
OllamaCUDA Graphs512-token micro-batches4k under 23 GiB, 32k at 23+ GiB, 256k at 47+ GiBFull available VRAMShared process; single-request default concurrency
HF TransformersEager PyTorchFull prompt in one passUnpaged DynamicCacheFull available VRAMShared process with layer split; static batching
GenericHardware-inferredHardware-inferredHardware-inferredFull available VRAMShared on consumer/Metal; per-GPU on datacenter

When Runtime Preset is set to Auto, the calculator chooses llama.cpp for Apple Silicon or GGUF quantizations on single consumer GPUs. It selects Ollama for standard quantizations on single NVIDIA consumer GPUs, and vLLM for datacenter GPUs or multi-GPU NVIDIA consumer rigs.

Diffusion execution engines

EngineExecution strategyMemory policyAuto-selection rule
Diffusers PipelineSequential pipeline with PyTorch SDPAPyTorch caching allocator; sequential sub-module CPU offload24 GB to 47 GB consumer GPUs and fine-tuning
ComfyUI Node EngineUnified node execution graphDynamic node-level weight swapping; zero cross-stage residual memoryConsumer GPUs under 24 GB VRAM
TensorRT DiffusionCaptured CUDA Graphs with static FMHA95% usable memory fraction; async multi-stream VAE decodeDatacenter GPUs or 48 GB+ cards

Worked example: serving a 70B FP8 model on a single 80 GB H100 under vLLM appears to leave 10 GB free when multiplying 70B by 1 byte. In practice, vLLM caps usable VRAM at 92% or 73.6 GB, while unquantized 16-bit embeddings, lm_head weights, driver context, and CUDA graph capture leave minimal KV block headroom for concurrent 32k-token requests. Inspect this configuration directly in the VRAM Calculator.

Fine-tuning frameworks

FrameworkAttention & lossCheckpointing & packingDefault optimizer & precisionAuto-selection rule
HF Transformers + PEFTPyTorch SDPA; full FP32 loss logitsCheckpointing off; dynamic padding32-bit AdamW; 16-bit embeddings under QLoRASingle-GPU consumer Full Fine-Tuning
UnslothFused Triton kernels; chunked cross-entropyHost-offloaded checkpointing; sequence packing onPaged 8-bit optimizerSingle-GPU LoRA and QLoRA
torchtuneFlash / SDPA; linear cross-entropy lossActivation checkpointing on; packing offPure BF16 weights and optimizer statesDatacenter Full Fine-Tuning and multi-GPU/multi-node training

Across <mNdp/> data-parallel GPUs, DeepSpeed ZeRO Stage 1 divides optimizer states by <mNdp/>. ZeRO Stage 2 divides optimizer states and gradients by <mNdp/>, and ZeRO Stage 3 or PyTorch FSDP divides parameters, gradients, optimizer states, and checkpointed activation boundaries by <mNdp/>.

Execution mode, prefill budgeting, and memory policy overrides

ControlOptionsPractical effect
Execution ModeGraph Captured vs. Dynamic EagerGraph capture eliminates Python/CPU dispatch overhead at the cost of static CUDA graph pool VRAM
Prefill ModeChunked 256 to 16,384 tokens, or Full SequenceChunking caps prefill activation spikes and prevents long prompts from stalling active decode streams
GPU Memory ReservedFraction from 0.50 to 0.98Sets how much VRAM the engine reserves for weights plus KV block pools before rejecting new requests
Prefix CachingOn or OffReuses KV blocks for shared system prompts, reducing per-user KV growth and TTFT
Continuous BatchingContinuous vs. StaticIteration-level scheduling improves aggregate throughput under uneven request lengths
Static VRAM Pre-allocationOn-Demand vs. Pre-allocated fractionModels engines that claim the full KV memory pool upfront at startup

Performance modeling, roofline dynamics & SLA knees

Throughput and latency follow physical hardware rooflines coupled with runtime scheduling behavior. The performance model covers single-stream decode speed, prompt prefill latency, and tail-latency cliffs as concurrent KV pools approach saturation.

Bandwidth vs. compute roofline & Time to First Token

Autoregressive decode throughput is bounded by hardware memory bandwidth <mBw/> and dense compute <mFlops/> scaled by calibrated silicon, quantization, and batch efficiency terms <mEtaBw/> and <mEtaCompute/>:

Normalized decode throughput vs. batch size as execution shifts from memory-bandwidth bound to compute bound

KV pool utilizationP50 TTFT multiplierP99 TTFT multiplier
20%1x1.2x
40%1.1x1.5x
60%1.3x2.1x
80%1.7x3.2x
95%2.4x5.1x
100%3.1x7.4x
110%6.8x18.5x

The transition from memory-bandwidth bound to compute-bound decode occurs at the arithmetic-intensity crossover batch size <mBstar/>:

For Mixture-of-Experts models, rising batch size <mB/> increases the count of distinct routed experts activated per step, shifting decode bandwidth demand smoothly from active-expert weights toward the full resident expert pool.

Prompt prefill is compute-bound across linear projections and quadratic <mSseq2/> self-attention at long contexts:

For Diffusion Transformers, each denoising step combines linear backbone FLOPs <mDitLinear/> with quadratic patch attention <mDitAttn/>.

Effective bandwidth, compute utilization, and dispatch overheads are calibrated against physical hardware benchmarks on the Accuracy Scorecard.

Concurrency scaling, TTFT latency cliffs, and safe operating knees

Remaining usable VRAM is divided into fixed-size KV blocks such as 16 tokens in vLLM or 32 tokens in TensorRT-LLM, with each user's unique non-shared tokens rounded up to whole blocks. As request load approaches theoretical service capacity, continuous-batching queue delay rises sharply and drives a tail-latency cliff at the Safe Operating Knee; sustainable QPS is bounded below that saturation knee.

Relative TTFT latency multiplier vs. KV block pool utilization across the safe operating knee

KV pool utilizationP50 TTFT multiplierP99 TTFT multiplier
20%1x1.2x
40%1.1x1.5x
60%1.3x2.1x
80%1.7x3.2x
95%2.4x5.1x
100%3.1x7.4x
110%6.8x18.5x

The concurrency analyzer tracks allocator admission occupancy watermarks for moderate and critical preemption risk, reports P50, P95, and P99 TTFT, and flags prefill starvation when unchunked long prompts stall active decode streams.

Sequential & block-parallel speculative decoding

Sequential drafting uses a smaller standalone model to propose <mGamma/> tokens verified in parallel by the target model. Block-parallel drafting such as DFlash, Medusa, and Eagle proposes a token block in a single pass.

Both methods raise low-batch decode speed, but resident draft weights shrink the free KV block pool and lower the preemption-free concurrency ceiling.

Step time decomposition, Tensor Core MFU, and activation headroom

Training step latency splits into four phases: Forward Pass, Backward Pass including gradient-checkpointing recomputation, Collective Comms across TP/PP/DP groups, and Optimizer Step including host offload transfers. ZeRO Stage 3 and FSDP model both All-Gather parameter and Reduce-Scatter gradient traffic across <mNdp/> devices, whereas ZeRO Stage 2 models Reduce-Scatter gradient traffic alone, flagging steps that become communication-bound.

Sweeping micro-batch sizes from 1 to 32 against remaining activation headroom plots Tensor Core MFU saturation, identifies the largest safe micro-batch before backward activations OOM, and computes the gradient accumulation steps required to reach your target global batch size.

Multi-GPU parallelism, cluster topologies & offloading

Multi-device setups span Single Rig workstations and multi-node Clusters partitioned across Tensor Parallelism, Pipeline Parallelism, Sequence Parallelism, and Data Parallelism.

Tensor, Pipeline, Sequence, and Data Parallelism

StrategyWhat is splitCommunication patternWhen to use
Tensor ParallelismAttention heads, MLP matrices, KV cacheAll-reduce on every layerLowest single-user latency inside an NVLink or xGMI node
Pipeline ParallelismLayer groups across stagesPoint-to-point stage handoffSpanning PCIe links or multiple nodes
Sequence Parallelism2D and 3D latent patch tokens in DiTRing or Ulysses attention exchangeHigh-resolution image and video diffusion
Data ParallelismIndependent replicas or ZeRO shardsNone in inference; ZeRO collectives in trainingScaling aggregate QPS or training batch size

Under Pipeline Parallelism across <mNpp/> stages, Stage 0 holds input embeddings and the Final Stage holds lm_head on top of their layer share. Outer stages OOM first if a cluster is sized by average per-GPU memory:

Per-stage weight memory for a 70B FP16 model with 128k untied vocabulary across 4 pipeline stages

StageTransformer layersEmbeddings / lm_head
Stage 035 GB2 GB
Stage 135 GB0 GB
Stage 235 GB0 GB
Stage 335 GB2 GB

In a 4-stage pipeline on a 70B FP16 model with a 128k untied vocabulary, middle stages hold 35.0 GB of layers while Stage 0 and Stage 3 hold 37.0 GB each. Sizing purely by the 35.5 GB average underestimates outer-stage peak demand by 1.5 GB per GPU. Evaluate multi-GPU splits in the VRAM Calculator.

Intra-node fabrics, inter-node networks & disaggregated serving

TierSupported interconnectsViable parallelism
High-Speed FabricsNVLink Gen 2 through Gen 6, NVLink Bridge, AMD Infinity Fabric xGMI, InfiniBand EDR through XDRTensor Parallelism, Sequence Parallelism, FSDP / ZeRO-3
PCIePCIe 3.0, 4.0, and 5.0 x16Pipeline Parallelism, Data Parallelism, low-degree TP
Network & External LinksThunderbolt 4 and 5, Ethernet 1G through 400GPipeline Parallelism or replicated Data Parallelism

When Tensor Parallelism, Pipeline Parallelism stages, or FSDP/ZeRO replicas span multiple physical nodes, communication efficiency is blended between the intra-node fabric and inter-node network according to the exact fraction of cross-node hops.

Multi-node configurations also output a Disaggregated Prefill/Decode Network Budget detailing per-request KV transfer volume, required fabric bandwidth, and minimum NIC line rate from 100G Ethernet to 800G InfiniBand / RoCE.

Intra-node high-speed fabric vs. inter-node network hops across TP, PP, and DP/ZeRO groups

Node 0 and Node 1 run intra-node TP and PP over NVLink or xGMI, while cross-node PP, DP/ZeRO, and KV transfers traverse InfiniBand or Ethernet.

Weight, KV cache, gradient, and optimizer offloading

Offload targetGranularityPerformance impact
Model WeightsLayer count or percentage to System RAM / NVMe; prioritizes MoE expert weights before shared layersShifts streamed layer reads from HBM to PCIe / NVMe
KV CacheFull KV pool to System RAMTransfers attention state over PCIe on every token, severely reducing decode tok/s
ZeRO Training StatesOptimizer states, gradients, and activations to CPU / NVMeAdds PCIe transfer latency to the optimizer and backward steps

Worked example: serving a 70B FP8 model on a single 80 GB H100 under vLLM appears to leave 10 GB free when multiplying 70B by 1 byte. In practice, vLLM caps usable VRAM at 92% or 73.6 GB, while unquantized 16-bit embeddings, lm_head weights, driver context, and CUDA graph capture leave minimal KV block headroom for concurrent 32k-token requests. Inspect this configuration directly in the VRAM Calculator.

Hardware calibration & accuracy verification

Predictions are continuously verified against physical hardware traces, with live test cases and error distributions published on the Accuracy Scorecard.

Ground-truth benchmark suites & evaluation metrics

SuiteWhat is verifiedGround-truth source
Suite A: Model WeightsStatic weight footprintsPublished .safetensors and GGUF artifact sizes
Suite B: KV CacheAttention KV scalingExact tensor geometry across MHA, GQA, MQA, and MLA
Suite C / N: Decode SpeedInference throughput and TTFTPhysical runs across Apple Silicon, consumer GPUs, and datacenter GPUs
Suite D: Fine-Tuning VRAMTraining peak memoryProfiler traces across ZeRO-3, LoRA, QLoRA, and Full Fine-Tuning
Suite E: End-to-End Inference VRAMTotal serving allocationtorch.cuda.max_memory_allocated and engine memory telemetry
Suite T: Training SpeedStep latency and tok/sMeasured runs across torchtune, HF FSDP, DeepSpeed, and NeMo

Each suite is scored by In-Band Pass Rate, Median Error, Geometric Mean Fold Error, and Systemic Bias. Representative ground-truth checks from the active suite:

Case IDConfigurationMeasuredPredicted
A1Llama-3-8B FP16 safetensors weights14.97 GB14.96 GB
A2Qwen2.5-32B Q4_K_M GGUF weights18.49 GB18.18 GB
E2Qwen2.5-32B Q4_K_M end-to-end 16k ctx24.00 GB23.20 GB
C3LLaMA-7B Q4_0 on M2 Ultra 76c, llama.cpp94.3 tok/s96.0 tok/s
D1Llama-3.1-405B FP8, 8x H200 SXM, TensorRT-LLM TP=856.0 tok/s55.0 tok/s

Modeling scope & error tolerances

Memory capacity predictions achieve sub-1% median error because tensor dimensions, KV block pages, optimizer states, and driver contexts follow deterministic physical bounds.

Throughput and training speed estimates carry wider variance, typically 14% to 24%, due to host CPU dispatch jitter, thermal and electrical throttling, background PCIe contention, and undocumented runtime flags.

For programmatic integration into CI/CD pipelines or autonomous agents, see the REST API Documentation and MCP Server Documentation.

Last updated: November 17, 2026