NVIDIA A10G (24GB)
The NVIDIA A10G (24GB) features 24 GB of dedicated VRAM and 600 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
24 GB
Usable Memory Ceiling
24 GB
Bandwidth
600 GB/s
Max Inference Class
32B Models (Q4)
TDP
300W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA A10G (24GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA A10G (24GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#48 | Qwen 3.8 27B | 27B | INT4 | 30 tok/s | ~4.8s | |
#83 | K2 Horizon MoVA 36B A4B | 36B (4B active) | INT4 | 122 tok/s | ~1.0s | |
#93 | Qwen3.5-27B | 27B | INT4 | 30 tok/s | ~4.8s | |
#105 | Sarvam-30B | 32B (2.4B active) | INT4 | 171 tok/s | ~652 ms | |
#106 | Qwen3.6 35B A3B | 35B (3B active) | INT4 | 144 tok/s | ~827 ms | |
#110 | Muse Glimmer 30B | 30B | INT4 | 27 tok/s | ~5.3s | |
#111 | MiMo V2 Flash | 15B (309B active) | INT4 Q8 | 3 tok/s 1 tok/s | ~52.2s ~57.0s | |
#112 | Phi-4 Reasoning Plus | 14B | INT4 Q8 | 39 tok/s 26 tok/s | ~2.5s ~2.5s | |
#117 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 | 138 tok/s | ~855 ms | |
#125 | Qwen3.5-9B | 9B | INT4 Q8 FP16 | 81 tok/s 48 tok/s 24 tok/s | ~1.6s ~1.6s ~1.8s | |
#126 | Qwen3.5-35B-A3B | 35B (3B active) | INT4 | 144 tok/s | ~827 ms | |
#128 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
#135 | Gemma 4 12B | 11.95B | INT4 Q8 | 45 tok/s 30 tok/s | ~2.2s ~2.2s | |
#140 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
#143 | GPT-OSS 20B | 21B (3.6B active) | INT4 | 63 tok/s | ~808 ms | |
#144 | GLM-4V | 9B | INT4 Q8 FP16 | 57 tok/s 39 tok/s 21 tok/s | ~1.6s ~1.6s ~1.8s | |
#150 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 26 tok/s 15 tok/s 8 tok/s | ~5.1s ~5.2s ~5.2s | |
#153 | Qwen3-30B-A3B | 30B (3B active) | INT4 | 154 tok/s | ~746 ms | |
#158 | Granite 4.2 8B | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#159 | Gemma 4 E4B | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#165 | GLM-4 | 32B | INT4 | 18 tok/s | ~6.6s | |
#166 | Phi-4 | 14B | INT4 Q8 | 54 tok/s 32 tok/s | ~2.5s ~2.5s | |
#167 | Gemma 3 27B | 27B | INT4 | 30 tok/s | ~4.8s | |
#170 | DeepSeek-R1 32B | 32B | INT4 | 26 tok/s | ~5.6s | |
#171 | Qwen3-32B | 32B | INT4 | 25 tok/s | ~5.6s | |
#175 | Magistral Small | 24B | INT4 | 25 tok/s | ~4.3s | |
#176 | OLMo 3.1 32B Instruct | 32B | INT4 | 18 tok/s | ~6.6s | |
#177 | Qwen2.5-32B | 32B | INT4 | 25 tok/s | ~5.6s | |
#178 | OLMo 3 7B Think | 7B | INT4 Q8 FP16 | 70 tok/s 49 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
#179 | DeepSeek-R1 14B | 14B | INT4 Q8 | 56 tok/s 32 tok/s | ~2.5s ~2.5s | |
#183 | Ministral 3 14B | 14B | INT4 Q8 | 39 tok/s 26 tok/s | ~2.5s ~2.5s | |
#186 | Gemma 3 12B | 12B | INT4 Q8 FP16 | 63 tok/s 37 tok/s 17 tok/s | ~2.2s ~2.2s ~2.6s | |
#188 | Mistral-Small-2501 | 24B | INT4 | 33 tok/s | ~4.3s | |
#189 | Ministral 3 8B | 8B | INT4 Q8 FP16 | 63 tok/s 44 tok/s 24 tok/s | ~1.5s ~1.5s ~1.6s | |
#190 | Gemma 4 31B | 30.7B | INT4 | 26 tok/s | ~5.4s | |
#191 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
#192 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
#193 | Gemma 2 27B | 27B | INT4 | 30 tok/s | ~4.8s | |
#194 | Qwen2.5-14B | 14B | INT4 Q8 | 54 tok/s 32 tok/s | ~2.5s ~2.5s | |
#195 | OLMo 3.1 32B Think | 32B | INT4 | 18 tok/s | ~6.6s | |
#196 | Nemotron 3.5 Lightning | 30B (3B active) | INT4 | 54 tok/s | ~822 ms | |
#200 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 128 tok/s 79 tok/s 43 tok/s | ~939 ms ~944 ms ~955 ms | |
#202 | DeepSeek-R1 8B | 8B | INT4 Q8 FP16 | 92 tok/s 54 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#204 | Qwen3-4B | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
#205 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 121 tok/s 92 tok/s 59 tok/s | ~574 ms ~576 ms ~583 ms | |
#206 | OLMo 3 32B Base | 32B | INT4 | 18 tok/s | ~6.6s | |
#207 | Gemma 2 9B | 9B | INT4 Q8 FP16 | 81 tok/s 48 tok/s 24 tok/s | ~1.6s ~1.6s ~1.8s | |
#208 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 390 tok/s 302 tok/s 201 tok/s | ~162 ms ~163 ms ~165 ms | |
#209 | Qwen3-14B | 14B | INT4 Q8 | 54 tok/s 32 tok/s | ~2.5s ~2.5s | |
#210 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
#211 | Qwen3-8B | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#213 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 102 tok/s 61 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
#214 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 266 tok/s 187 tok/s 112 tok/s | ~340 ms ~341 ms ~345 ms | |
#215 | Llama 3.1 8B | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#216 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 473 tok/s 426 tok/s 344 tok/s | ~55 ms ~55 ms ~56 ms | |
#217 | Command R | 35B | INT4 | 16 tok/s | ~7.2s | |
#218 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 437 tok/s 351 tok/s 244 tok/s | ~120 ms ~121 ms ~122 ms | |
#219 | Llama 3 8B | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#221 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 160 tok/s 102 tok/s 56 tok/s | ~711 ms ~715 ms ~723 ms | |
#223 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
#224 | Ministral-8B-2410 | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#225 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 286 tok/s 204 tok/s 125 tok/s | ~301 ms ~302 ms ~305 ms | |
#226 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
#227 | Nemotron 3.5 Lightning 30B A3B | 30B (3B active) | INT4 | 154 tok/s | ~746 ms | |
#228 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 160 tok/s 102 tok/s 56 tok/s | ~711 ms ~715 ms ~723 ms | |
#229 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 121 tok/s 102 tok/s 72 tok/s | ~428 ms ~429 ms ~433 ms | |
#230 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 96 tok/s 58 tok/s 31 tok/s | ~1.3s ~1.3s ~1.3s | |
#231 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 350 tok/s 264 tok/s 170 tok/s | ~206 ms ~207 ms ~209 ms | |
#232 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
#233 | Phi-3-medium | 14B | INT4 Q8 | 54 tok/s 32 tok/s | ~2.5s ~2.5s | |
#234 | OLMo 3 7B Instruct | 7B | INT4 Q8 FP16 | 70 tok/s 49 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
#235 | Yi-34B | 34B | INT4 | 17 tok/s | ~7.0s | |
#236 | Qwen2-7B | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
#237 | Phi-3-small | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
#238 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 350 tok/s 264 tok/s 170 tok/s | ~206 ms ~207 ms ~209 ms | |
#239 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 96 tok/s 58 tok/s 31 tok/s | ~1.3s ~1.3s ~1.3s | |
#241 | Gemma 1 7B | 7B | INT4 Q8 FP16 | 70 tok/s 49 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
#242 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 259 tok/s 174 tok/s 101 tok/s | ~397 ms ~398 ms ~403 ms | |
#243 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 78 tok/s 55 tok/s 33 tok/s | ~1.1s ~1.1s ~1.1s | |
#244 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 78 tok/s 55 tok/s 33 tok/s | ~1.1s ~1.1s ~1.1s | |
#245 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 78 tok/s 55 tok/s 33 tok/s | ~1.1s ~1.1s ~1.1s | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 531 tok/s 462 tok/s 358 tok/s | ~61 ms ~61 ms ~62 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 465 tok/s 382 tok/s 273 tok/s | ~100 ms ~101 ms ~102 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 465 tok/s 382 tok/s 273 tok/s | ~100 ms ~101 ms ~102 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 454 tok/s 407 tok/s 324 tok/s | ~61 ms ~61 ms ~62 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 454 tok/s 407 tok/s 324 tok/s | ~61 ms ~61 ms ~62 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 437 tok/s 351 tok/s 244 tok/s | ~120 ms ~121 ms ~122 ms | |
- | PixArt-α XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 435 tok/s 348 tok/s 242 tok/s | ~122 ms ~123 ms ~124 ms | |
- | PixArt-Σ XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 435 tok/s 348 tok/s 242 tok/s | ~122 ms ~123 ms ~124 ms | |
- | Nemotron 3.5 Parse | 0.63B | INT4 Q8 FP16 | 429 tok/s 342 tok/s 236 tok/s | ~127 ms ~128 ms ~129 ms | |
- | SD 2.1 Turbo | 0.86B | INT4 Q8 FP16 | 377 tok/s 289 tok/s 191 tok/s | ~175 ms ~176 ms ~177 ms | |
- | Stable Diffusion 1.4 | 0.86B | INT4 Q8 FP16 | 377 tok/s 289 tok/s 191 tok/s | ~175 ms ~176 ms ~177 ms | |
- | Stable Diffusion 2.1 | 0.86B | INT4 Q8 FP16 | 377 tok/s 289 tok/s 191 tok/s | ~175 ms ~176 ms ~177 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 372 tok/s 275 tok/s 175 tok/s | ~206 ms ~207 ms ~209 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 350 tok/s 264 tok/s 170 tok/s | ~206 ms ~207 ms ~209 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 350 tok/s 264 tok/s 170 tok/s | ~206 ms ~207 ms ~209 ms | |
- | Kandinsky 2.2 Decoder | 1.22B | INT4 Q8 FP16 | 318 tok/s 234 tok/s 147 tok/s | ~248 ms ~249 ms ~251 ms | |
- | Wan 2.1 T2V 1.3B | 1.3B | INT4 Q8 FP16 | 309 tok/s 225 tok/s 140 tok/s | ~262 ms ~263 ms ~266 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 286 tok/s 204 tok/s 125 tok/s | ~301 ms ~302 ms ~305 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 286 tok/s 204 tok/s 125 tok/s | ~301 ms ~302 ms ~305 ms | |
- | SVD Image-to-Video XT | 1.5B | INT4 Q8 FP16 | 286 tok/s 204 tok/s 125 tok/s | ~301 ms ~302 ms ~305 ms | |
- | SVD Image-to-Video XT 1.1 | 1.5B | INT4 Q8 FP16 | 286 tok/s 204 tok/s 125 tok/s | ~301 ms ~302 ms ~305 ms | |
- | LFM3 1.7B | 1.7B | INT4 Q8 FP16 | 266 tok/s 187 tok/s 112 tok/s | ~340 ms ~341 ms ~345 ms | |
- | LFM3 1.7B Base | 1.7B | INT4 Q8 FP16 | 266 tok/s 187 tok/s 112 tok/s | ~340 ms ~341 ms ~345 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 266 tok/s 187 tok/s 112 tok/s | ~340 ms ~341 ms ~345 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 258 tok/s 180 tok/s 107 tok/s | ~358 ms ~359 ms ~363 ms | |
- | Moondream 3.1 | 9B (2B active) | INT4 Q8 FP16 | 242 tok/s 166 tok/s 93 tok/s | ~447 ms ~449 ms ~492 ms | |
- | CogVideoX 2B | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
- | LFM3 Audio 2B | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
- | LFM3 Omni 2B | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
- | LFM3 VL 2B | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
- | Liquid D1 2B | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
- | LTX-Video 0.9.1 | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
- | Stable Diffusion 3 Medium | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
- | Kandinsky 3.0 | 2.1B | INT4 Q8 FP16 | 235 tok/s 161 tok/s 94 tok/s | ~414 ms ~416 ms ~421 ms | |
- | LTX-2 | 2.2B | INT4 Q8 FP16 | 228 tok/s 155 tok/s 91 tok/s | ~432 ms ~434 ms ~438 ms | |
- | SDXL 1.0 Refiner | 2.31B | INT4 Q8 FP16 | 222 tok/s 150 tok/s 87 tok/s | ~451 ms ~453 ms ~458 ms | |
- | Stable Diffusion 3.5 Medium | 2.5B | INT4 Q8 FP16 | 211 tok/s 141 tok/s 81 tok/s | ~484 ms ~486 ms ~491 ms | |
- | SDXL 1.0 Base | 2.56B | INT4 Q8 FP16 | 208 tok/s 139 tok/s 80 tok/s | ~494 ms ~497 ms ~502 ms | |
- | SDXL Turbo | 2.56B | INT4 Q8 FP16 | 208 tok/s 139 tok/s 80 tok/s | ~494 ms ~497 ms ~502 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 200 tok/s 128 tok/s 71 tok/s | ~571 ms ~573 ms ~580 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 196 tok/s 162 tok/s 113 tok/s | ~264 ms ~265 ms ~268 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 196 tok/s 162 tok/s 113 tok/s | ~264 ms ~265 ms ~268 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 196 tok/s 162 tok/s 113 tok/s | ~264 ms ~265 ms ~268 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 191 tok/s 124 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 Q8 FP16 | 191 tok/s 133 tok/s 71 tok/s | ~558 ms ~560 ms ~664 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | LFM2.5 24B A2B | 24B (2.5B active) | INT4 | 175 tok/s | ~632 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 Q8 | 162 tok/s 101 tok/s | ~698 ms ~824 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 Q8 | 162 tok/s 101 tok/s | ~698 ms ~824 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 | 156 tok/s | ~734 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 | 156 tok/s | ~734 ms | |
- | FLUX.2 Klein 4B | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
- | Hy-Memory 30B A3B | 30B (3B active) | INT4 | 154 tok/s | ~746 ms | |
- | Hy-MT2 30B A3B | 30B (3B active) | INT4 | 154 tok/s | ~746 ms | |
- | Mage-VL Cowork | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 147 tok/s 118 tok/s 79 tok/s | ~399 ms ~401 ms ~405 ms | |
- | MiniCPM-V 4.7 35B A3B | 35B (3B active) | INT4 | 144 tok/s | ~827 ms | |
- | CogVideoX 1.5 5B | 5B | INT4 Q8 FP16 | 130 tok/s 81 tok/s 44 tok/s | ~922 ms ~927 ms ~937 ms | |
- | CogVideoX 5B | 5B | INT4 Q8 FP16 | 130 tok/s 81 tok/s 44 tok/s | ~922 ms ~927 ms ~937 ms | |
- | Wan 2.2 TI2V 5B | 5B | INT4 Q8 FP16 | 130 tok/s 81 tok/s 44 tok/s | ~922 ms ~927 ms ~937 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 128 tok/s 99 tok/s 64 tok/s | ~521 ms ~524 ms ~529 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 121 tok/s 92 tok/s 59 tok/s | ~574 ms ~576 ms ~583 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 121 tok/s 92 tok/s 59 tok/s | ~574 ms ~576 ms ~583 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 104 tok/s 62 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
- | AuraFlow v0.2 | 6.8B | INT4 Q8 FP16 | 102 tok/s 62 tok/s 33 tok/s | ~1.2s ~1.2s ~1.3s | |
- | AuraFlow v0.3 | 6.8B | INT4 Q8 FP16 | 102 tok/s 62 tok/s 33 tok/s | ~1.2s ~1.2s ~1.3s | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 96 tok/s 58 tok/s 31 tok/s | ~1.3s ~1.3s ~1.3s | |
- | Youtu Parsing Omni | 8B | INT4 Q8 FP16 | 92 tok/s 54 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
- | Stable Diffusion 3.5 Large | 8.1B | INT4 Q8 FP16 | 88 tok/s 53 tok/s 28 tok/s | ~1.5s ~1.5s ~1.5s | |
- | Stable Diffusion 3.5 Large Turbo | 8.1B | INT4 Q8 FP16 | 88 tok/s 53 tok/s 28 tok/s | ~1.5s ~1.5s ~1.5s | |
- | FLUX.2 Klein 9B | 9B | INT4 Q8 FP16 | 81 tok/s 48 tok/s 24 tok/s | ~1.6s ~1.6s ~1.8s | |
- | LensVLM 9B | 9B | INT4 Q8 FP16 | 81 tok/s 48 tok/s 24 tok/s | ~1.6s ~1.6s ~1.8s | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 FP16 | 81 tok/s 48 tok/s 24 tok/s | ~1.6s ~1.6s ~1.8s | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 78 tok/s 55 tok/s 33 tok/s | ~1.1s ~1.1s ~1.1s | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 78 tok/s 55 tok/s 33 tok/s | ~1.1s ~1.1s ~1.1s | |
- | Falcon3-10B | 10B | INT4 Q8 FP16 | 74 tok/s 43 tok/s 22 tok/s | ~1.8s ~1.8s ~2.0s | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 Q8 | 74 tok/s 60 tok/s | ~677 ms ~738 ms | |
- | Mochi 1 Preview | 10B | INT4 Q8 FP16 | 74 tok/s 43 tok/s 22 tok/s | ~1.8s ~1.8s ~2.0s | |
- | Falcon2-11B | 11B | INT4 Q8 FP16 | 71 tok/s 41 tok/s 19 tok/s | ~2.0s ~2.0s ~2.4s | |
- | Hunyuan Lite | 7B | INT4 Q8 FP16 | 70 tok/s 49 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
- | OLMo 3 7B Base | 7B | INT4 Q8 FP16 | 70 tok/s 49 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
- | SEA-LION-7B | 7.1B | INT4 Q8 FP16 | 69 tok/s 48 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 FP16 | 69 tok/s 48 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
- | FLUX.1 Canny Dev | 11.9B | INT4 Q8 FP16 | 63 tok/s 37 tok/s 17 tok/s | ~2.1s ~2.2s ~2.6s | |
- | FLUX.1 Depth Dev | 11.9B | INT4 Q8 FP16 | 63 tok/s 37 tok/s 17 tok/s | ~2.1s ~2.2s ~2.6s | |
- | FLUX.1 Dev | 11.9B | INT4 Q8 FP16 | 63 tok/s 37 tok/s 17 tok/s | ~2.1s ~2.2s ~2.6s | |
- | FLUX.1 Fill Dev | 11.9B | INT4 Q8 FP16 | 63 tok/s 37 tok/s 17 tok/s | ~2.1s ~2.2s ~2.6s | |
- | FLUX.1 Schnell | 11.9B | INT4 Q8 FP16 | 63 tok/s 37 tok/s 17 tok/s | ~2.1s ~2.2s ~2.6s | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 FP16 | 63 tok/s 44 tok/s 24 tok/s | ~1.5s ~1.5s ~1.6s | |
- | Typhoon-2-8B | 8B | INT4 Q8 FP16 | 63 tok/s 44 tok/s 24 tok/s | ~1.5s ~1.5s ~1.6s | |
- | HunyuanVideo 1.5 720P T2V | 13B | INT4 Q8 | 58 tok/s 34 tok/s | ~2.3s ~2.3s | |
- | GLM-4-9B | 9B | INT4 Q8 FP16 | 57 tok/s 39 tok/s 21 tok/s | ~1.6s ~1.6s ~1.8s | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 FP16 | 57 tok/s 39 tok/s 21 tok/s | ~1.6s ~1.6s ~1.8s | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 FP16 | 57 tok/s 39 tok/s 21 tok/s | ~1.6s ~1.6s ~1.8s | |
- | Yi-9B | 9B | INT4 Q8 FP16 | 57 tok/s 39 tok/s 21 tok/s | ~1.6s ~1.6s ~1.8s | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 FP16 | 56 tok/s 38 tok/s 21 tok/s | ~1.7s ~1.7s ~1.8s | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 FP16 | 56 tok/s 38 tok/s 21 tok/s | ~1.7s ~1.7s ~1.8s | |
- | Wan 2.1 I2V 14B 720P | 14B | INT4 Q8 | 54 tok/s 32 tok/s | ~2.5s ~2.5s | |
- | Wan 2.1 T2V 14B | 14B | INT4 Q8 | 54 tok/s 32 tok/s | ~2.5s ~2.5s | |
- | Wan 2.2 T2V A14B | 14B | INT4 Q8 | 54 tok/s 32 tok/s | ~2.5s ~2.5s | |
- | FLUX.2 Dev | 15B | INT4 Q8 | 51 tok/s 30 tok/s | ~2.7s ~2.7s | |
- | GreenMind-14B-R1 | 14B | INT4 Q8 | 39 tok/s 26 tok/s | ~2.5s ~2.5s | |
- | Arete 27B | 27B | INT4 | 30 tok/s | ~4.8s | |
- | AREX-2 | 27B | INT4 | 30 tok/s | ~4.8s | |
- | PPLX Decider v1 27B | 27B | INT4 | 30 tok/s | ~4.8s | |
- | PPLX Decider v1.1 27B | 27B | INT4 | 30 tok/s | ~4.8s | |
- | Agnes 3.0 Flash | 33B | INT4 | 25 tok/s | ~5.8s | |
- | Falcon-40B | 40B | INT4 | 20 tok/s | ~7.6s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
32B Models
8-bit
14B Models
16-bit
8B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
32B Models
LoRA
14B Models
Full Parameter Training
3B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA A10G (24GB).
Inference Sweet Spot
Optimized for 14B models at uncompressed or Q8 precision, and 32B models at Q4 with up to 32k context.
Bandwidth & Speed Profile
With 600 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 133 tok/s on an 8B Q4 model and 16 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning