REST API DOCUMENTATION

VRAM Calculator API

Integrate programmatic LLM model requirements checks into your Dashboard, CI/CD workflows, or custom deployment scripts.

API Reference

Authentication

Calculations require an API token created in settings. Include it as a Bearer credential in the request header.

Authorization: Bearer YOUR_API_KEY_HERE
Generate Token
POST
/api/v1/vram-calculation/inference

Computes required GPU memory allocation breakdown and performance metrics specifically for LLM inference workloads.

Request JSON Body Parameters: Inference

FieldTypeDescription & Enum Choices

Core Model and Hardware Parameters

gpu_key *stringGPU profile key, such as 4090_24 or h100_80. See GET /gpus for a full list.
llm_key *stringModel variant slug, such as llama-3-8b-instruct or custom. See GET /llms for a full list.
quantizationstringWeights quantization target. Choices: fp32, fp16, fp8, int8, int4, q8, q4_k_m, etc. Default is fp16.
custom_vramnumberCustom GPU memory capacity in GB. Required if gpu_key is custom_discrete or custom_apple_silicon.
custom_bandwidthnumberCustom GPU memory bandwidth in GB/s. Required if gpu_key is custom_discrete or custom_apple_silicon.
custom_tflopsnumberCustom GPU FP16 compute performance in TFLOPS. Required if gpu_key is custom_discrete or custom_apple_silicon.
framework_presetstringRuntime the estimate models: auto, vllm, tensorrt_llm, llama_cpp, ollama, hf_transformers, diffusers or generic. Default is auto, which picks llama_cpp for GGUF quantizations, diffusers for diffusion architectures, and vllm otherwise. generic reproduces the framework-agnostic estimate of earlier API versions with legacy_auto accepted as an alias. The response's framework_preset field names the runtime used.

Language Model Workload Parameters

batch_sizeintegerBatch size. Default is 1.
seq_lengthintegerContext sequence length in tokens. Default is 2048.
concurrent_usersintegerNumber of concurrent users. Default is 1.
kv_cache_quantizationstringKV cache sharding/precision. Choices: fp16, fp8, int8, int4. Default is fp16.
enable_prefix_cachingbooleanEnable KV cache prefix caching. Defaults to the framework preset's behavior, which is on for vllm and tensorrt_llm. Takes effect with shared_prefix_ratio above 0.
shared_prefix_rationumberFraction of prompt sequence that is a shared prefix. Range: 0.0 to 1.0. Default is 0.0.
enable_continuous_batchingbooleanEnable continuous batching. Defaults to the framework preset's behavior.
inference_parallelismstringParallelism strategy. Choices: pipeline, tensor. Default is pipeline.
vocab_sizeintegerCustom vocabulary size, such as 128256 for Llama 3 or 256000 for Gemma. Default is 32000.
ffn_intermediate_sizeintegerCustom MLP or SwiGLU intermediate dimension size, such as 14336 for Llama 3 8B.
kv_source_layer_countintegerNumber of layers generating shared KV cache states for compressed or hybrid architectures like CSA2.
kv_latent_dimintegerLatent compressed KV cache dimension for architectures like MLA or CSA2.
indexer_dimintegerSparse routing indexer key dimension for architectures like DSA or CSA2.

Diffusion and DiT Workload Parameters

image_widthintegerTarget image width in pixels for diffusion generation. Defaults to 1024 for images and 1280 for video.
image_heightintegerTarget image height in pixels for diffusion generation. Defaults to 1024 for images and 720 for video.
num_framesintegerTotal frames generated for video diffusion models. Defaults to 1 for images and 49 for video.
inference_stepsintegerDenoising inference step count for diffusion generation. Default is 20.
guidance_scalenumberClassifier-free guidance scale value. Default is 1.0 for Schnell and 3.5 for Dev.
encoder_offloadbooleanSequentially offload text encoders like T5 or CLIP to host CPU memory during denoising steps. Default is true.
vae_tilingbooleanEnable spatial and temporal tiled VAE decoding to bound peak activation VRAM. Default is false for images and true for video.

Distributed Hardware and Topology Parameters

num_gpusintegerNumber of GPUs distributed on a single node. Defaults to 1 when cluster parameters are not used; mutually exclusive with num_nodes and gpus_per_node.
num_nodesintegerNumber of physical server chassis/nodes in the cluster. Required when using cluster configuration; mutually exclusive with num_gpus.
gpus_per_nodeintegerNumber of GPUs populated per server node in the cluster. Required when using cluster configuration; mutually exclusive with num_gpus.
tp_degreeintegerTensor Parallelism degree shards to partition weights across GPUs. Default is 1.
pp_degreeintegerPipeline Parallelism degree shards to distribute model layers sequentially across nodes or devices. Default is 1.
interconnect_typestringInterconnect type for multi-GPU setups. Supported values include PCIe types pcie3, pcie4, pcie5; NVLink types nvlink_bridge, nvlink_gen2 to nvlink_gen6; InfiniBand types infiniband_hdr, infiniband_edr; and Ethernet types ethernet10g to ethernet400g. Default is pcie4.
inter_node_interconnectstringNetwork connection interconnect between different nodes in the cluster. E.g., ethernet100g, ethernet400g, infiniband_hdr.

Host and Storage Offloading Parameters

enable_offloadingbooleanEnable GPU offloading. Default is false.
offload_targetstringStorage target for offloaded layers. Choices: cpu_ram, nvme.
num_offload_layersintegerNumber of model layers to offload.
percentage_offloadnumberPercentage of layers to offload, from 0 to 100.
offload_kv_cachebooleanOffload KV Cache to storage target. Default is false.

cURL Example: Inference

Shell Request

curl -X POST https://apxml.com/api/v1/vram-calculation/inference \
  -H "Authorization: Bearer YOUR_API_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{
    "gpu_key": "4090_24",
    "llm_key": "llama-3-8b-instruct",
    "quantization": "q4_k_m",
    "batch_size": 1,
    "seq_length": 2048
  }'

Python Example: Inference

Python Script

import requests

url = "https://apxml.com/api/v1/vram-calculation/inference"
headers = {
    "Authorization": "Bearer YOUR_API_KEY_HERE",
    "Content-Type": "application/json"
}
payload = {
    "gpu_key": "4090_24",
    "llm_key": "llama-3-8b-instruct",
    "quantization": "q4_k_m",
    "batch_size": 1,
    "seq_length": 2048
}

response = requests.post(url, headers=headers, json=payload)
print(response.json())

cURL Example: Diffusion Inference

Shell Request

curl -X POST https://apxml.com/api/v1/vram-calculation/inference \
  -H "Authorization: Bearer YOUR_API_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{
    "gpu_key": "4090_24",
    "llm_key": "flux-1-schnell",
    "quantization": "fp16",
    "image_width": 1024,
    "image_height": 1024,
    "encoder_offload": true
  }'

Python Example: Diffusion Inference

Python Script

import requests

url = "https://apxml.com/api/v1/vram-calculation/inference"
headers = {
    "Authorization": "Bearer YOUR_API_KEY_HERE",
    "Content-Type": "application/json"
}
payload = {
    "gpu_key": "4090_24",
    "llm_key": "flux-1-schnell",
    "quantization": "fp16",
    "image_width": 1024,
    "image_height": 1024,
    "encoder_offload": True
}

response = requests.post(url, headers=headers, json=payload)
print(response.json())
POST
/api/v1/vram-calculation/finetuning

Computes required GPU memory allocation breakdown, optimization impacts, and dataset training speeds for LLM fine-tuning and training workloads.

Request JSON Body Parameters: Fine-tuning

FieldTypeDescription & Enum Choices
gpu_key *stringGPU profile key, such as 4090_24 or h100_80. See GET /gpus for a full list.
llm_key *stringModel variant slug, such as llama-3-8b-instruct or custom. See GET /llms for a full list.
finetuning_methodstringFinetuning method. Choices: full, lora, qlora. Default is lora.
fine_tuning_quantizationstringBase model precision for full finetuning. Used when method is full. Choices: fp32, fp16, fp8. Default is fp16.
lora_rankintegerLoRA rank size. Used when method is lora or qlora. Default is 16.
gradient_accumulation_stepsintegerGradient accumulation steps. Default is 1.
batch_sizeintegerGlobal batch size. Default is 1.
seq_lengthintegerContext sequence length in tokens. Default is 2048.
num_samplesintegerDataset size in total samples.
tokens_per_sampleintegerAverage tokens per sample. Defaults to seq_length.
num_epochsintegerNumber of training epochs. Default is 3.
framework_presetstringTraining framework the estimate models: auto, hf_trainer, unsloth, torchtune or generic. Default is auto, which picks hf_trainer. generic reproduces the framework-agnostic estimate of earlier API versions with legacy_auto accepted as an alias. The response's framework_preset field names the framework used.
optimization_configobjectAdvanced optimization flags object. Contains boolean flags like flash_attention, gradient_checkpointing, use_8bit_optimizer, use_paged_optimizer, use_fused_kernels, activation_offloading, etc., and integer zero_stage.
custom_vramnumberCustom GPU memory capacity in GB. Required if gpu_key is custom_discrete or custom_apple_silicon.
custom_bandwidthnumberCustom GPU memory bandwidth in GB/s. Required if gpu_key is custom_discrete or custom_apple_silicon.
custom_tflopsnumberCustom GPU FP16 compute performance in TFLOPS. Required if gpu_key is custom_discrete or custom_apple_silicon.
num_gpusintegerNumber of GPUs distributed on a single node. Defaults to 1 when cluster parameters are not used; mutually exclusive with num_nodes and gpus_per_node.
num_nodesintegerNumber of physical server chassis/nodes in the cluster. Required when using cluster configuration; mutually exclusive with num_gpus.
gpus_per_nodeintegerNumber of GPUs populated per server node in the cluster. Required when using cluster configuration; mutually exclusive with num_gpus.
enable_offloadingbooleanEnable offloading to host system. Default is false.
offload_targetstringStorage target for offloaded layers. Choices: cpu_ram, nvme.
interconnect_typestringInterconnect type for multi-GPU setups. Supported values include PCIe types pcie3, pcie4, pcie5; NVLink types nvlink_bridge, nvlink_gen2 to nvlink_gen6; InfiniBand types infiniband_hdr, infiniband_edr; and Ethernet types ethernet10g to ethernet400g. Default is pcie4.
tp_degreeintegerTensor Parallelism degree shards to partition weights across GPUs. Default is 1.
pp_degreeintegerPipeline Parallelism degree shards to distribute model layers sequentially across nodes or devices. Default is 1.
inter_node_interconnectstringNetwork connection interconnect between different nodes in the cluster. E.g., ethernet100g, ethernet400g, infiniband_hdr.

cURL Example: Fine-tuning

Shell Request

curl -X POST https://apxml.com/api/v1/vram-calculation/finetuning \
  -H "Authorization: Bearer YOUR_API_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{
    "gpu_key": "4090_24",
    "llm_key": "llama-3-8b-instruct",
    "finetuning_method": "lora",
    "lora_rank": 16,
    "batch_size": 4,
    "seq_length": 2048
  }'

Python Example: Fine-tuning

Python Script

import requests

url = "https://apxml.com/api/v1/vram-calculation/finetuning"
headers = {
    "Authorization": "Bearer YOUR_API_KEY_HERE",
    "Content-Type": "application/json"
}
payload = {
    "gpu_key": "4090_24",
    "llm_key": "llama-3-8b-instruct",
    "finetuning_method": "lora",
    "lora_rank": 16,
    "batch_size": 4,
    "seq_length": 2048
}

response = requests.post(url, headers=headers, json=payload)
print(response.json())

Response Body: Inference Example

Inference Output

{
  "vram_usage": 6.84,
  "vram_percentage": 28.5,
  "actual_vram_percentage": 28.5,
  "memory_status": "okay",
  "static_shared_memory": 5.43,
  "per_user_memory": 0.52,
  "estimated_latency_tps": 45.2,
  "estimated_throughput_tps": 45.2,
  "per_user_tps": 45.2,
  "ms_per_token": 22.1,
  "ttft": 120,
  "ttft_ms": 120,
  "estimated_system_ram_required": 16.0,
  "offloaded_memory": 0.0,
  "estimated_power_draw": 320,
  "memory_details": {
    "Base Model Weights": 5.43,
    "KV Cache": 0.52,
    "Activations": 0.35,
    "Framework Overhead": 0.54
  },
  "memory_breakdown": [
    {
      "label": "Base Model Weights",
      "value": 79.4,
      "size_gb": 5.43
    },
    {
      "label": "KV Cache",
      "value": 7.6,
      "size_gb": 0.52
    }
  ],
  "runtime_advisories": [
    {
      "code": "PREFILL_STARVATION_RISK",
      "severity": "warning",
      "message": "Full-prompt prefill on long contexts stalls continuous batching token generation, causing high P99 latency jitter."
    }
  ]
};

Response Body: Fine-tuning Example

Fine-tuning Output

{
  "vram_usage": 32.4,
  "vram_percentage": 40.5,
  "actual_vram_percentage": 40.5,
  "memory_status": "okay",
  "static_shared_memory": 24.5,
  "training_tps": 12.0,
  "samples_per_second": 1.5,
  "steps_per_second": 0.18,
  "total_tokens": 10000000.0,
  "total_training_time_hours": 4.5,
  "estimated_system_ram_required": 32.0,
  "offloaded_memory": 0.0,
  "estimated_power_draw": 650,
  "active_optimizations": [
    "gradient_checkpointing",
    "flash_attention"
  ],
  "optimization_preset": "custom",
  "memory_details": {
    "Base Model Weights": 24.5,
    "Optimizer States": 29.4,
    "Gradients": 10.8
  },
  "memory_breakdown": [
    {
      "label": "Base Model Weights",
      "value": 38.2,
      "size_gb": 24.5
    },
    {
      "label": "Optimizer States",
      "value": 45.1,
      "size_gb": 29.4
    },
    {
      "label": "Gradients",
      "value": 16.7,
      "size_gb": 10.8
    }
  ]
}

Memory Status Enum Options

The memory_status field returns one of the following lowercase enum keys:

Enum ValueDescription
okayFits with safe margin representing 50% or less of VRAM
moderateFits with moderate usage representing 75% or less of VRAM
highTight fit representing 90% or less of VRAM
very_highVery close to capacity, exceeding 90% of VRAM
insufficientExceeds available VRAM capacity
errorGPU configuration or sizing calculation error

Runtime diagnostics and cluster metrics

The response includes diagnostic advisories and multi-node topology metrics when applicable:

FieldTypeDescription
runtime_advisoriesarray of objectsArray of advisory objects containing code, severity, and message, evaluating conditions such as memory allocator churn, KV cache swapping penalties, and hardware bottlenecks.
pipeline_asymmetryoptional objectStage memory imbalance metrics including stage_0_vram_gb, final_stage_vram_gb, and outer_stage_imbalance_ratio returned when pipeline parallelism pp_degree is greater than 1.
disaggregated_network_budgetoptional objectCross-node KV transfer requirements including kv_transfer_size_per_request_gb, required_bandwidth_gbs, and min_nic_line_rate returned for multi-node deployments when num_nodes is greater than 1.

Runtime advisory codes

The runtime_advisories array returns structured performance bottlenecks and stability advisories:

Advisory CodeSeverityTrigger Condition and Operational Impact

Language Model Advisories

PCIE_KV_SWAP_HAZARD
warning
Offloading KV cache to system RAM transfers attention state over PCIe on every token, significantly reducing generation speed.
PREFILL_STARVATION_RISK
warning
Full prompt prefill on long contexts stalls continuous batching across concurrent requests, increasing tail latency.
FP8_HARDWARE_EMULATION
info
Selected GPU unpacks FP8 weights at runtime, reducing generation speed.
INT8_DEQUANTIZATION_STALL
warning
INT8 weights dequantize on each step under eager execution, reducing generation speed.

Diffusion and Video Advisories

VAE_UNTILED_OOM_HAZARD
warning
Untiled video decoding causes peak memory spikes during the final decode step. Enable VAE tiling to prevent out of memory errors.

Distributed and Parallelism Advisories

INTER_NODE_TP_BOTTLENECK
warning
Tensor parallelism spans across physical nodes, introducing inter-node network bottlenecks on every token.

Deprecated Advisories

PREFIX_CACHE_ALIGNMENT
advisory

Sequence length is not aligned to the runtime KV block size. Unaligned prefix tokens cannot be shared.

Deprecated: Unaligned token overhead is negligible under paged memory allocation.

SPECULATIVE_KV_REDUCTION
advisory

Draft model VRAM footprint reduces remaining KV cache capacity and lowers concurrency saturation knee.

Deprecated: Use details_gb.draft_model and concurrency_sla for draft overhead.

PIPELINE_STAGE_ASYMMETRY
warning

Outer stages require more memory due to embedding and projection layers. Sizing must account for stage memory imbalance.

Deprecated: Use the pipeline_asymmetry response object for stage-by-stage memory breakdown.

Hardware & Model Listing Routes

GET
/api/v1/vram-calculation/gpus

Retrieve valid gpu_key parameters, hardware models, and VRAM capacities.

curl https://apxml.com/api/v1/vram-calculation/gpus \
  -H "Authorization: Bearer YOUR_API_KEY_HERE"
[
  {
    "key": "4090_24",
    "label": "NVIDIA GeForce RTX 4090",
    "memory": 24
  },
  {
    "key": "h100_80",
    "label": "NVIDIA H100 80GB",
    "memory": 80
  }
]
GET
/api/v1/vram-calculation/llms

Retrieve supported llm_key parameters mapped to model variant configurations. Supports query pagination parameters such as page and limit, for example ?page=1&limit=100.

curl https://apxml.com/api/v1/vram-calculation/llms \
  -H "Authorization: Bearer YOUR_API_KEY_HERE"
[
  {
    "key": "llama-3-8b-instruct",
    "name": "Llama 3 8B Instruct"
  },
  {
    "key": "llama-3.1-70b-instruct",
    "name": "Llama 3.1 70B Instruct"
  }
]