NVIDIA RTX PRO 4000 Blackwell SFF (24GB)
The NVIDIA RTX PRO 4000 Blackwell SFF (24GB) features 24 GB of dedicated VRAM and 432 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
24 GB
Usable Memory Ceiling
24 GB
Bandwidth
432 GB/s
Max Inference Class
32B Models (Q4)
TDP
70W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA RTX PRO 4000 Blackwell SFF (24GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA RTX PRO 4000 Blackwell SFF (24GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#48 | Qwen 3.8 27B | 27B | INT4 | 22 tok/s | ~3.1s | |
#83 | K2 Horizon MoVA 36B A4B | 36B (4B active) | INT4 | 87 tok/s | ~659 ms | |
#93 | Qwen3.5-27B | 27B | INT4 | 22 tok/s | ~3.1s | |
#105 | Sarvam-30B | 32B (2.4B active) | INT4 | 119 tok/s | ~424 ms | |
#106 | Qwen3.6 35B A3B | 35B (3B active) | INT4 | 102 tok/s | ~537 ms | |
#110 | Muse Glimmer 30B | 30B | INT4 | 20 tok/s | ~3.4s | |
#111 | MiMo V2 Flash | 15B (309B active) | INT4 Q8 | 2 tok/s 1 tok/s | ~33.6s ~36.9s | |
#112 | Phi-4 Reasoning Plus | 14B | INT4 Q8 | 30 tok/s 20 tok/s | ~1.6s ~1.6s | |
#117 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 | 98 tok/s | ~555 ms | |
#125 | Qwen3.5-9B | 9B | INT4 Q8 FP16 | 59 tok/s 36 tok/s 18 tok/s | ~1.1s ~1.1s ~1.2s | |
#126 | Qwen3.5-35B-A3B | 35B (3B active) | INT4 | 102 tok/s | ~537 ms | |
#128 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 72 tok/s 44 tok/s 24 tok/s | ~825 ms ~833 ms ~852 ms | |
#135 | Gemma 4 12B | 11.95B | INT4 Q8 | 34 tok/s 23 tok/s | ~1.4s ~1.4s | |
#140 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 108 tok/s 71 tok/s 40 tok/s | ~486 ms ~491 ms ~502 ms | |
#143 | GPT-OSS 20B | 21B (3.6B active) | INT4 | 47 tok/s | ~531 ms | |
#144 | GLM-4V | 9B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 16 tok/s | ~1.1s ~1.1s ~1.2s | |
#150 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 19 tok/s 11 tok/s 6 tok/s | ~3.3s ~3.3s ~3.4s | |
#153 | Qwen3-30B-A3B | 30B (3B active) | INT4 | 108 tok/s | ~485 ms | |
#158 | Granite 4.2 8B | 8B | INT4 Q8 FP16 | 65 tok/s 40 tok/s 21 tok/s | ~937 ms ~947 ms ~969 ms | |
#159 | Gemma 4 E4B | 8B | INT4 Q8 FP16 | 65 tok/s 40 tok/s 21 tok/s | ~937 ms ~947 ms ~969 ms | |
#165 | GLM-4 | 32B | INT4 | 13 tok/s | ~4.3s | |
#166 | Phi-4 | 14B | INT4 Q8 | 40 tok/s 24 tok/s | ~1.6s ~1.6s | |
#167 | Gemma 3 27B | 27B | INT4 | 22 tok/s | ~3.1s | |
#170 | DeepSeek-R1 32B | 32B | INT4 | 20 tok/s | ~3.6s | |
#171 | Qwen3-32B | 32B | INT4 | 19 tok/s | ~3.6s | |
#175 | Magistral Small | 24B | INT4 | 19 tok/s | ~2.8s | |
#176 | OLMo 3.1 32B Instruct | 32B | INT4 | 13 tok/s | ~4.3s | |
#177 | Qwen2.5-32B | 32B | INT4 | 19 tok/s | ~3.6s | |
#178 | OLMo 3 7B Think | 7B | INT4 Q8 FP16 | 52 tok/s 36 tok/s 21 tok/s | ~830 ms ~838 ms ~857 ms | |
#179 | DeepSeek-R1 14B | 14B | INT4 Q8 | 42 tok/s 24 tok/s | ~1.6s ~1.6s | |
#183 | Ministral 3 14B | 14B | INT4 Q8 | 30 tok/s 20 tok/s | ~1.6s ~1.6s | |
#186 | Gemma 3 12B | 12B | INT4 Q8 FP16 | 46 tok/s 27 tok/s 13 tok/s | ~1.4s ~1.4s ~1.7s | |
#188 | Mistral-Small-2501 | 24B | INT4 | 25 tok/s | ~2.7s | |
#189 | Ministral 3 8B | 8B | INT4 Q8 FP16 | 47 tok/s 32 tok/s 18 tok/s | ~943 ms ~953 ms ~1.1s | |
#190 | Gemma 4 31B | 30.7B | INT4 | 20 tok/s | ~3.5s | |
#191 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 162 tok/s 116 tok/s 71 tok/s | ~261 ms ~263 ms ~269 ms | |
#192 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 162 tok/s 116 tok/s 71 tok/s | ~261 ms ~263 ms ~269 ms | |
#193 | Gemma 2 27B | 27B | INT4 | 22 tok/s | ~3.1s | |
#194 | Qwen2.5-14B | 14B | INT4 Q8 | 40 tok/s 24 tok/s | ~1.6s ~1.6s | |
#195 | OLMo 3.1 32B Think | 32B | INT4 | 13 tok/s | ~4.3s | |
#196 | Nemotron 3.5 Lightning | 30B (3B active) | INT4 | 40 tok/s | ~542 ms | |
#200 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 92 tok/s 58 tok/s 32 tok/s | ~610 ms ~616 ms ~630 ms | |
#202 | DeepSeek-R1 8B | 8B | INT4 Q8 FP16 | 67 tok/s 40 tok/s 21 tok/s | ~937 ms ~947 ms ~969 ms | |
#204 | Qwen3-4B | 4B | INT4 Q8 FP16 | 108 tok/s 71 tok/s 40 tok/s | ~486 ms ~491 ms ~502 ms | |
#205 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 87 tok/s 67 tok/s 43 tok/s | ~377 ms ~380 ms ~388 ms | |
#206 | OLMo 3 32B Base | 32B | INT4 | 13 tok/s | ~4.3s | |
#207 | Gemma 2 9B | 9B | INT4 Q8 FP16 | 59 tok/s 36 tok/s 18 tok/s | ~1.1s ~1.1s ~1.2s | |
#208 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 244 tok/s 197 tok/s 138 tok/s | ~110 ms ~111 ms ~113 ms | |
#209 | Qwen3-14B | 14B | INT4 Q8 | 40 tok/s 24 tok/s | ~1.6s ~1.6s | |
#210 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 108 tok/s 71 tok/s 40 tok/s | ~486 ms ~491 ms ~502 ms | |
#211 | Qwen3-8B | 8B | INT4 Q8 FP16 | 65 tok/s 40 tok/s 21 tok/s | ~937 ms ~947 ms ~969 ms | |
#213 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 74 tok/s 45 tok/s 24 tok/s | ~824 ms ~833 ms ~852 ms | |
#214 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 81 tok/s | ~224 ms ~226 ms ~231 ms | |
#215 | Llama 3.1 8B | 8B | INT4 Q8 FP16 | 65 tok/s 40 tok/s 21 tok/s | ~937 ms ~947 ms ~969 ms | |
#216 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 285 tok/s 263 tok/s 220 tok/s | ~41 ms ~41 ms ~42 ms | |
#217 | Command R | 35B | INT4 | 12 tok/s | ~4.7s | |
#218 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 268 tok/s 224 tok/s 164 tok/s | ~83 ms ~84 ms ~85 ms | |
#219 | Llama 3 8B | 8B | INT4 Q8 FP16 | 65 tok/s 40 tok/s 21 tok/s | ~937 ms ~947 ms ~969 ms | |
#221 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 112 tok/s 74 tok/s 42 tok/s | ~463 ms ~468 ms ~478 ms | |
#223 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 72 tok/s 44 tok/s 24 tok/s | ~825 ms ~833 ms ~852 ms | |
#224 | Ministral-8B-2410 | 8B | INT4 Q8 FP16 | 65 tok/s 40 tok/s 21 tok/s | ~937 ms ~947 ms ~969 ms | |
#225 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 188 tok/s 140 tok/s 89 tok/s | ~199 ms ~201 ms ~205 ms | |
#226 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 130 tok/s 88 tok/s 51 tok/s | ~373 ms ~377 ms ~385 ms | |
#227 | Nemotron 3.5 Lightning 30B A3B | 30B (3B active) | INT4 | 108 tok/s | ~485 ms | |
#228 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 112 tok/s 74 tok/s 42 tok/s | ~463 ms ~468 ms ~478 ms | |
#229 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 87 tok/s 74 tok/s 53 tok/s | ~283 ms ~285 ms ~291 ms | |
#230 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 70 tok/s 43 tok/s 23 tok/s | ~858 ms ~867 ms ~887 ms | |
#231 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 223 tok/s 176 tok/s 119 tok/s | ~138 ms ~139 ms ~142 ms | |
#232 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 162 tok/s 116 tok/s 71 tok/s | ~261 ms ~263 ms ~269 ms | |
#233 | Phi-3-medium | 14B | INT4 Q8 | 40 tok/s 24 tok/s | ~1.6s ~1.6s | |
#234 | OLMo 3 7B Instruct | 7B | INT4 Q8 FP16 | 52 tok/s 36 tok/s 21 tok/s | ~830 ms ~838 ms ~857 ms | |
#235 | Yi-34B | 34B | INT4 | 13 tok/s | ~4.5s | |
#236 | Qwen2-7B | 7B | INT4 Q8 FP16 | 72 tok/s 44 tok/s 24 tok/s | ~825 ms ~833 ms ~852 ms | |
#237 | Phi-3-small | 7B | INT4 Q8 FP16 | 72 tok/s 44 tok/s 24 tok/s | ~825 ms ~833 ms ~852 ms | |
#238 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 223 tok/s 176 tok/s 119 tok/s | ~138 ms ~139 ms ~142 ms | |
#239 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 70 tok/s 43 tok/s 23 tok/s | ~858 ms ~867 ms ~887 ms | |
#241 | Gemma 1 7B | 7B | INT4 Q8 FP16 | 52 tok/s 36 tok/s 21 tok/s | ~830 ms ~838 ms ~857 ms | |
#242 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 173 tok/s 121 tok/s 73 tok/s | ~260 ms ~263 ms ~268 ms | |
#243 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 57 tok/s 41 tok/s 25 tok/s | ~717 ms ~724 ms ~740 ms | |
#244 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 57 tok/s 41 tok/s 25 tok/s | ~717 ms ~724 ms ~740 ms | |
#245 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 57 tok/s 41 tok/s 25 tok/s | ~717 ms ~724 ms ~740 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 312 tok/s 280 tok/s 228 tok/s | ~45 ms ~45 ms ~46 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 282 tok/s 240 tok/s 181 tok/s | ~70 ms ~70 ms ~72 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 282 tok/s 240 tok/s 181 tok/s | ~70 ms ~70 ms ~72 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 277 tok/s 253 tok/s 210 tok/s | ~45 ms ~45 ms ~46 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 277 tok/s 253 tok/s 210 tok/s | ~45 ms ~45 ms ~46 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 268 tok/s 224 tok/s 164 tok/s | ~83 ms ~84 ms ~85 ms | |
- | PixArt-α XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 267 tok/s 223 tok/s 163 tok/s | ~84 ms ~85 ms ~87 ms | |
- | PixArt-Σ XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 267 tok/s 223 tok/s 163 tok/s | ~84 ms ~85 ms ~87 ms | |
- | Nemotron 3.5 Parse | 0.63B | INT4 Q8 FP16 | 264 tok/s 219 tok/s 159 tok/s | ~87 ms ~88 ms ~90 ms | |
- | SD 2.1 Turbo | 0.86B | INT4 Q8 FP16 | 238 tok/s 190 tok/s 132 tok/s | ~118 ms ~119 ms ~121 ms | |
- | Stable Diffusion 1.4 | 0.86B | INT4 Q8 FP16 | 238 tok/s 190 tok/s 132 tok/s | ~118 ms ~119 ms ~121 ms | |
- | Stable Diffusion 2.1 | 0.86B | INT4 Q8 FP16 | 238 tok/s 190 tok/s 132 tok/s | ~118 ms ~119 ms ~121 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 235 tok/s 182 tok/s 122 tok/s | ~138 ms ~139 ms ~142 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 223 tok/s 176 tok/s 119 tok/s | ~138 ms ~139 ms ~142 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 223 tok/s 176 tok/s 119 tok/s | ~138 ms ~139 ms ~142 ms | |
- | Kandinsky 2.2 Decoder | 1.22B | INT4 Q8 FP16 | 206 tok/s 158 tok/s 104 tok/s | ~165 ms ~166 ms ~170 ms | |
- | Wan 2.1 T2V 1.3B | 1.3B | INT4 Q8 FP16 | 201 tok/s 152 tok/s 99 tok/s | ~174 ms ~176 ms ~179 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 188 tok/s 140 tok/s 89 tok/s | ~199 ms ~201 ms ~205 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 188 tok/s 140 tok/s 89 tok/s | ~199 ms ~201 ms ~205 ms | |
- | SVD Image-to-Video XT | 1.5B | INT4 Q8 FP16 | 188 tok/s 140 tok/s 89 tok/s | ~199 ms ~201 ms ~205 ms | |
- | SVD Image-to-Video XT 1.1 | 1.5B | INT4 Q8 FP16 | 188 tok/s 140 tok/s 89 tok/s | ~199 ms ~201 ms ~205 ms | |
- | LFM3 1.7B | 1.7B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 81 tok/s | ~224 ms ~226 ms ~231 ms | |
- | LFM3 1.7B Base | 1.7B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 81 tok/s | ~224 ms ~226 ms ~231 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 81 tok/s | ~224 ms ~226 ms ~231 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 172 tok/s 125 tok/s 77 tok/s | ~235 ms ~238 ms ~243 ms | |
- | Moondream 3.1 | 9B (2B active) | INT4 Q8 FP16 | 163 tok/s 116 tok/s 68 tok/s | ~293 ms ~295 ms ~326 ms | |
- | CogVideoX 2B | 2B | INT4 Q8 FP16 | 162 tok/s 116 tok/s 71 tok/s | ~261 ms ~263 ms ~269 ms | |
- | LFM3 Audio 2B | 2B | INT4 Q8 FP16 | 162 tok/s 116 tok/s 71 tok/s | ~261 ms ~263 ms ~269 ms | |
- | LFM3 Omni 2B | 2B | INT4 Q8 FP16 | 162 tok/s 116 tok/s 71 tok/s | ~261 ms ~263 ms ~269 ms | |
- | LFM3 VL 2B | 2B | INT4 Q8 FP16 | 162 tok/s 116 tok/s 71 tok/s | ~261 ms ~263 ms ~269 ms | |
- | Liquid D1 2B | 2B | INT4 Q8 FP16 | 162 tok/s 116 tok/s 71 tok/s | ~261 ms ~263 ms ~269 ms | |
- | LTX-Video 0.9.1 | 2B | INT4 Q8 FP16 | 162 tok/s 116 tok/s 71 tok/s | ~261 ms ~263 ms ~269 ms | |
- | Stable Diffusion 3 Medium | 2B | INT4 Q8 FP16 | 162 tok/s 116 tok/s 71 tok/s | ~261 ms ~263 ms ~269 ms | |
- | Kandinsky 3.0 | 2.1B | INT4 Q8 FP16 | 158 tok/s 113 tok/s 68 tok/s | ~272 ms ~275 ms ~280 ms | |
- | LTX-2 | 2.2B | INT4 Q8 FP16 | 155 tok/s 109 tok/s 66 tok/s | ~283 ms ~286 ms ~292 ms | |
- | SDXL 1.0 Refiner | 2.31B | INT4 Q8 FP16 | 151 tok/s 106 tok/s 63 tok/s | ~295 ms ~298 ms ~305 ms | |
- | Stable Diffusion 3.5 Medium | 2.5B | INT4 Q8 FP16 | 144 tok/s 100 tok/s 60 tok/s | ~317 ms ~320 ms ~327 ms | |
- | SDXL 1.0 Base | 2.56B | INT4 Q8 FP16 | 143 tok/s 98 tok/s 58 tok/s | ~323 ms ~327 ms ~334 ms | |
- | SDXL Turbo | 2.56B | INT4 Q8 FP16 | 143 tok/s 98 tok/s 58 tok/s | ~323 ms ~327 ms ~334 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 138 tok/s 91 tok/s 52 tok/s | ~372 ms ~376 ms ~384 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 135 tok/s 114 tok/s 81 tok/s | ~176 ms ~178 ms ~181 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 135 tok/s 114 tok/s 81 tok/s | ~176 ms ~178 ms ~181 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 135 tok/s 114 tok/s 81 tok/s | ~176 ms ~178 ms ~181 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 132 tok/s 89 tok/s 51 tok/s | ~373 ms ~376 ms ~385 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 Q8 FP16 | 132 tok/s 94 tok/s 52 tok/s | ~364 ms ~367 ms ~438 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 130 tok/s 88 tok/s 51 tok/s | ~373 ms ~377 ms ~385 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 130 tok/s 88 tok/s 51 tok/s | ~373 ms ~377 ms ~385 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 130 tok/s 88 tok/s 51 tok/s | ~373 ms ~377 ms ~385 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 130 tok/s 88 tok/s 51 tok/s | ~373 ms ~377 ms ~385 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 130 tok/s 88 tok/s 51 tok/s | ~373 ms ~377 ms ~385 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 130 tok/s 88 tok/s 51 tok/s | ~373 ms ~377 ms ~385 ms | |
- | LFM2.5 24B A2B | 24B (2.5B active) | INT4 | 122 tok/s | ~412 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 Q8 | 114 tok/s 73 tok/s | ~454 ms ~537 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 Q8 | 114 tok/s 73 tok/s | ~454 ms ~537 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 | 110 tok/s | ~478 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 | 110 tok/s | ~478 ms | |
- | FLUX.2 Klein 4B | 4B | INT4 Q8 FP16 | 108 tok/s 71 tok/s 40 tok/s | ~486 ms ~491 ms ~502 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 108 tok/s 71 tok/s 40 tok/s | ~486 ms ~491 ms ~502 ms | |
- | Hy-Memory 30B A3B | 30B (3B active) | INT4 | 108 tok/s | ~485 ms | |
- | Hy-MT2 30B A3B | 30B (3B active) | INT4 | 108 tok/s | ~485 ms | |
- | Mage-VL Cowork | 4B | INT4 Q8 FP16 | 108 tok/s 71 tok/s 40 tok/s | ~486 ms ~491 ms ~502 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 108 tok/s 71 tok/s 40 tok/s | ~486 ms ~491 ms ~502 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 104 tok/s 85 tok/s 58 tok/s | ~264 ms ~266 ms ~272 ms | |
- | MiniCPM-V 4.7 35B A3B | 35B (3B active) | INT4 | 102 tok/s | ~537 ms | |
- | CogVideoX 1.5 5B | 5B | INT4 Q8 FP16 | 93 tok/s 59 tok/s 33 tok/s | ~599 ms ~605 ms ~618 ms | |
- | CogVideoX 5B | 5B | INT4 Q8 FP16 | 93 tok/s 59 tok/s 33 tok/s | ~599 ms ~605 ms ~618 ms | |
- | Wan 2.2 TI2V 5B | 5B | INT4 Q8 FP16 | 93 tok/s 59 tok/s 33 tok/s | ~599 ms ~605 ms ~618 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 91 tok/s 72 tok/s 47 tok/s | ~343 ms ~346 ms ~353 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 87 tok/s 67 tok/s 43 tok/s | ~377 ms ~380 ms ~388 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 87 tok/s 67 tok/s 43 tok/s | ~377 ms ~380 ms ~388 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 75 tok/s 46 tok/s 24 tok/s | ~824 ms ~833 ms ~852 ms | |
- | AuraFlow v0.2 | 6.8B | INT4 Q8 FP16 | 74 tok/s 46 tok/s 25 tok/s | ~802 ms ~810 ms ~829 ms | |
- | AuraFlow v0.3 | 6.8B | INT4 Q8 FP16 | 74 tok/s 46 tok/s 25 tok/s | ~802 ms ~810 ms ~829 ms | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 72 tok/s 44 tok/s 24 tok/s | ~825 ms ~833 ms ~852 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 72 tok/s 44 tok/s 24 tok/s | ~825 ms ~833 ms ~852 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 70 tok/s 43 tok/s 23 tok/s | ~858 ms ~867 ms ~887 ms | |
- | Youtu Parsing Omni | 8B | INT4 Q8 FP16 | 67 tok/s 40 tok/s 21 tok/s | ~937 ms ~947 ms ~969 ms | |
- | Stable Diffusion 3.5 Large | 8.1B | INT4 Q8 FP16 | 64 tok/s 39 tok/s 21 tok/s | ~948 ms ~958 ms ~980 ms | |
- | Stable Diffusion 3.5 Large Turbo | 8.1B | INT4 Q8 FP16 | 64 tok/s 39 tok/s 21 tok/s | ~948 ms ~958 ms ~980 ms | |
- | FLUX.2 Klein 9B | 9B | INT4 Q8 FP16 | 59 tok/s 36 tok/s 18 tok/s | ~1.1s ~1.1s ~1.2s | |
- | LensVLM 9B | 9B | INT4 Q8 FP16 | 59 tok/s 36 tok/s 18 tok/s | ~1.1s ~1.1s ~1.2s | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 FP16 | 59 tok/s 36 tok/s 18 tok/s | ~1.1s ~1.1s ~1.2s | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 57 tok/s 41 tok/s 25 tok/s | ~717 ms ~724 ms ~740 ms | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 57 tok/s 41 tok/s 25 tok/s | ~717 ms ~724 ms ~740 ms | |
- | Falcon3-10B | 10B | INT4 Q8 FP16 | 54 tok/s 32 tok/s 16 tok/s | ~1.2s ~1.2s ~1.3s | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 Q8 | 54 tok/s 45 tok/s | ~446 ms ~487 ms | |
- | Mochi 1 Preview | 10B | INT4 Q8 FP16 | 54 tok/s 32 tok/s 16 tok/s | ~1.2s ~1.2s ~1.3s | |
- | Falcon2-11B | 11B | INT4 Q8 FP16 | 52 tok/s 30 tok/s 14 tok/s | ~1.3s ~1.3s ~1.5s | |
- | Hunyuan Lite | 7B | INT4 Q8 FP16 | 52 tok/s 36 tok/s 21 tok/s | ~830 ms ~838 ms ~857 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 FP16 | 52 tok/s 36 tok/s 21 tok/s | ~830 ms ~838 ms ~857 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 FP16 | 51 tok/s 36 tok/s 21 tok/s | ~841 ms ~850 ms ~869 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 FP16 | 51 tok/s 36 tok/s 21 tok/s | ~841 ms ~850 ms ~869 ms | |
- | FLUX.1 Canny Dev | 11.9B | INT4 Q8 FP16 | 47 tok/s 28 tok/s 13 tok/s | ~1.4s ~1.4s ~1.7s | |
- | FLUX.1 Depth Dev | 11.9B | INT4 Q8 FP16 | 47 tok/s 28 tok/s 13 tok/s | ~1.4s ~1.4s ~1.7s | |
- | FLUX.1 Dev | 11.9B | INT4 Q8 FP16 | 47 tok/s 28 tok/s 13 tok/s | ~1.4s ~1.4s ~1.7s | |
- | FLUX.1 Fill Dev | 11.9B | INT4 Q8 FP16 | 47 tok/s 28 tok/s 13 tok/s | ~1.4s ~1.4s ~1.7s | |
- | FLUX.1 Schnell | 11.9B | INT4 Q8 FP16 | 47 tok/s 28 tok/s 13 tok/s | ~1.4s ~1.4s ~1.7s | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 FP16 | 47 tok/s 32 tok/s 18 tok/s | ~943 ms ~953 ms ~1.1s | |
- | Typhoon-2-8B | 8B | INT4 Q8 FP16 | 47 tok/s 32 tok/s 18 tok/s | ~943 ms ~953 ms ~1.1s | |
- | HunyuanVideo 1.5 720P T2V | 13B | INT4 Q8 | 43 tok/s 25 tok/s | ~1.5s ~1.5s | |
- | GLM-4-9B | 9B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 16 tok/s | ~1.1s ~1.1s ~1.2s | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 16 tok/s | ~1.1s ~1.1s ~1.2s | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 16 tok/s | ~1.1s ~1.1s ~1.2s | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 16 tok/s | ~1.1s ~1.1s ~1.2s | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 16 tok/s | ~1.1s ~1.1s ~1.2s | |
- | Yi-9B | 9B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 16 tok/s | ~1.1s ~1.1s ~1.2s | |
- | Wan 2.1 I2V 14B 720P | 14B | INT4 Q8 | 40 tok/s 24 tok/s | ~1.6s ~1.6s | |
- | Wan 2.1 T2V 14B | 14B | INT4 Q8 | 40 tok/s 24 tok/s | ~1.6s ~1.6s | |
- | Wan 2.2 T2V A14B | 14B | INT4 Q8 | 40 tok/s 24 tok/s | ~1.6s ~1.6s | |
- | FLUX.2 Dev | 15B | INT4 Q8 | 38 tok/s 22 tok/s | ~1.7s ~1.8s | |
- | GreenMind-14B-R1 | 14B | INT4 Q8 | 30 tok/s 20 tok/s | ~1.6s ~1.6s | |
- | Arete 27B | 27B | INT4 | 22 tok/s | ~3.1s | |
- | AREX-2 | 27B | INT4 | 22 tok/s | ~3.1s | |
- | PPLX Decider v1 27B | 27B | INT4 | 22 tok/s | ~3.1s | |
- | PPLX Decider v1.1 27B | 27B | INT4 | 22 tok/s | ~3.1s | |
- | Agnes 3.0 Flash | 33B | INT4 | 19 tok/s | ~3.7s | |
- | Falcon-40B | 40B | INT4 | 15 tok/s | ~4.9s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
32B Models
8-bit
14B Models
16-bit
8B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
32B Models
LoRA
14B Models
Full Parameter Training
3B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA RTX PRO 4000 Blackwell SFF (24GB).
Inference Sweet Spot
Optimized for 14B models at uncompressed or Q8 precision, and 32B models at Q4 with up to 32k context.
Bandwidth & Speed Profile
With 432 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 96 tok/s on an 8B Q4 model and 11 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning