NVIDIA L20 (48GB)
The NVIDIA L20 (48GB) features 48 GB of dedicated VRAM and 864 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
48 GB
Usable Memory Ceiling
48 GB
Bandwidth
864 GB/s
Max Inference Class
32B Models (Q4)
TDP
275W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA L20 (48GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA L20 (48GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#48 | Qwen 3.8 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
#83 | K2 Horizon MoVA 36B A4B | 36B (4B active) | INT4 Q8 | 171 tok/s 119 tok/s | ~550 ms ~553 ms | |
#93 | Qwen3.5-27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
#105 | Sarvam-30B | 32B (2.4B active) | INT4 Q8 | 221 tok/s 166 tok/s | ~384 ms ~385 ms | |
#106 | Qwen3.6 35B A3B | 35B (3B active) | INT4 Q8 | 199 tok/s 144 tok/s | ~448 ms ~450 ms | |
#107 | Nemotron 3.5 Bolt 74B A9B | 74B | INT4 | 15 tok/s | ~8.3s | |
#110 | Muse Glimmer 30B | 30B | INT4 Q8 | 38 tok/s 22 tok/s | ~3.1s ~3.1s | |
#111 | MiMo V2 Flash | 15B (309B active) | INT4 Q8 FP16 | 4 tok/s 2 tok/s 1 tok/s | ~30.6s ~30.8s ~31.3s | |
#112 | Phi-4 Reasoning Plus | 14B | INT4 Q8 FP16 | 55 tok/s 37 tok/s 21 tok/s | ~1.5s ~1.5s ~1.5s | |
#117 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 Q8 | 183 tok/s 126 tok/s | ~503 ms ~506 ms | |
#125 | Qwen3.5-9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~956 ms ~962 ms ~975 ms | |
#126 | Qwen3.5-35B-A3B | 35B (3B active) | INT4 Q8 | 199 tok/s 144 tok/s | ~448 ms ~450 ms | |
#128 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
#135 | Gemma 4 12B | 11.95B | INT4 Q8 FP16 | 63 tok/s 43 tok/s 25 tok/s | ~1.3s ~1.3s ~1.3s | |
#140 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
#143 | GPT-OSS 20B | 21B (3.6B active) | INT4 Q8 FP16 | 88 tok/s 76 tok/s 49 tok/s | ~477 ms ~478 ms ~567 ms | |
#144 | GLM-4V | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~959 ms ~965 ms ~978 ms | |
#150 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 36 tok/s 21 tok/s 11 tok/s | ~3.0s ~3.0s ~3.1s | |
#153 | Qwen3-30B-A3B | 30B (3B active) | INT4 Q8 | 201 tok/s 146 tok/s | ~439 ms ~441 ms | |
#158 | Granite 4.2 8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#159 | Gemma 4 E4B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#162 | Qwen2.5-72B | 72B | INT4 | 16 tok/s | ~8.0s | |
#163 | Mistral Large 3 | 41B (675B active) | INT4 | 2 tok/s | ~66.9s | |
#164 | DeepSeek-R1 70B | 70B | INT4 | 18 tok/s | ~7.2s | |
#165 | GLM-4 | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~3.3s ~3.6s | |
#166 | Phi-4 | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~1.5s ~1.5s ~1.5s | |
#167 | Gemma 3 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
#168 | Llama 3.3 70B | 70B | INT4 | 17 tok/s | ~7.2s | |
#170 | DeepSeek-R1 32B | 32B | INT4 Q8 | 37 tok/s 21 tok/s | ~3.3s ~3.3s | |
#171 | Qwen3-32B | 32B | INT4 Q8 | 36 tok/s 20 tok/s | ~3.3s ~3.3s | |
#173 | Hunyuan Standard | 52B (389B active) | INT4 | 3 tok/s | ~38.6s | |
#175 | Magistral Small | 24B | INT4 Q8 | 36 tok/s 23 tok/s | ~2.5s ~2.5s | |
#176 | OLMo 3.1 32B Instruct | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~3.3s ~3.6s | |
#177 | Qwen2.5-32B | 32B | INT4 Q8 | 36 tok/s 20 tok/s | ~3.3s ~3.3s | |
#178 | OLMo 3 7B Think | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~751 ms ~755 ms ~766 ms | |
#179 | DeepSeek-R1 14B | 14B | INT4 Q8 FP16 | 78 tok/s 45 tok/s 24 tok/s | ~1.5s ~1.5s ~1.5s | |
#181 | Llama 3.1 70B | 70B | INT4 | 17 tok/s | ~7.2s | |
#183 | Ministral 3 14B | 14B | INT4 Q8 FP16 | 55 tok/s 37 tok/s 21 tok/s | ~1.5s ~1.5s ~1.5s | |
#186 | Gemma 3 12B | 12B | INT4 Q8 FP16 | 87 tok/s 51 tok/s 27 tok/s | ~1.3s ~1.3s ~1.3s | |
#187 | Qwen2-72B | 72B | INT4 | 16 tok/s | ~8.0s | |
#188 | Mistral-Small-2501 | 24B | INT4 Q8 FP16 | 47 tok/s 27 tok/s 13 tok/s | ~2.5s ~2.5s ~3.0s | |
#189 | Ministral 3 8B | 8B | INT4 Q8 FP16 | 88 tok/s 61 tok/s 36 tok/s | ~854 ms ~859 ms ~871 ms | |
#190 | Gemma 4 31B | 30.7B | INT4 Q8 | 37 tok/s 21 tok/s | ~3.2s ~3.2s | |
#191 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
#192 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
#193 | Gemma 2 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
#194 | Qwen2.5-14B | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~1.5s ~1.5s ~1.5s | |
#195 | OLMo 3.1 32B Think | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~3.3s ~3.6s | |
#196 | Nemotron 3.5 Lightning | 30B (3B active) | INT4 Q8 | 79 tok/s 71 tok/s | ~447 ms ~448 ms | |
#198 | Llama 3 70B | 70B | INT4 | 17 tok/s | ~7.2s | |
#200 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 170 tok/s 109 tok/s 60 tok/s | ~553 ms ~556 ms ~563 ms | |
#202 | DeepSeek-R1 8B | 8B | INT4 Q8 FP16 | 124 tok/s 75 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#203 | Kimi Linear 48B A3B Instruct | 48B (3B active) | INT4 | 67 tok/s | ~484 ms | |
#204 | Qwen3-4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
#205 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 162 tok/s 125 tok/s 81 tok/s | ~338 ms ~340 ms ~345 ms | |
#206 | OLMo 3 32B Base | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~3.3s ~3.6s | |
#207 | Gemma 2 9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~956 ms ~962 ms ~975 ms | |
#208 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 446 tok/s 362 tok/s 255 tok/s | ~97 ms ~97 ms ~98 ms | |
#209 | Qwen3-14B | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~1.5s ~1.5s ~1.5s | |
#210 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
#211 | Qwen3-8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#213 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 138 tok/s 84 tok/s 46 tok/s | ~748 ms ~753 ms ~763 ms | |
#214 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 326 tok/s 240 tok/s 151 tok/s | ~201 ms ~202 ms ~204 ms | |
#215 | Llama 3.1 8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#216 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 519 tok/s 479 tok/s 403 tok/s | ~34 ms ~34 ms ~34 ms | |
#217 | Command R | 35B | INT4 Q8 | 26 tok/s 16 tok/s | ~3.6s ~4.0s | |
#218 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 489 tok/s 411 tok/s 303 tok/s | ~72 ms ~72 ms ~73 ms | |
#219 | Llama 3 8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#221 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 208 tok/s 137 tok/s 78 tok/s | ~419 ms ~421 ms ~427 ms | |
#222 | Mixtral-8x7B-v0.1 | 46.7B (7B active) | INT4 Q8 | 119 tok/s 70 tok/s | ~869 ms ~1.0s | |
#223 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
#224 | Ministral-8B-2410 | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#225 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 346 tok/s 259 tok/s 166 tok/s | ~178 ms ~179 ms ~181 ms | |
#226 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
#227 | Nemotron 3.5 Lightning 30B A3B | 30B (3B active) | INT4 Q8 | 201 tok/s 146 tok/s | ~439 ms ~441 ms | |
#228 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 208 tok/s 137 tok/s 78 tok/s | ~419 ms ~421 ms ~427 ms | |
#229 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 162 tok/s 138 tok/s 99 tok/s | ~253 ms ~254 ms ~257 ms | |
#230 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 130 tok/s 80 tok/s 43 tok/s | ~779 ms ~783 ms ~794 ms | |
#231 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 409 tok/s 323 tok/s 220 tok/s | ~122 ms ~123 ms ~124 ms | |
#232 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
#233 | Phi-3-medium | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~1.5s ~1.5s ~1.5s | |
#234 | OLMo 3 7B Instruct | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~751 ms ~755 ms ~766 ms | |
#235 | Yi-34B | 34B | INT4 Q8 | 27 tok/s 16 tok/s | ~3.5s ~3.8s | |
#236 | Qwen2-7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
#237 | Phi-3-small | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
#238 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 409 tok/s 323 tok/s 220 tok/s | ~122 ms ~123 ms ~124 ms | |
#239 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 130 tok/s 80 tok/s 43 tok/s | ~779 ms ~783 ms ~794 ms | |
#241 | Gemma 1 7B | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~751 ms ~755 ms ~766 ms | |
#242 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 319 tok/s 225 tok/s 136 tok/s | ~234 ms ~235 ms ~238 ms | |
#243 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~648 ms ~652 ms ~660 ms | |
#244 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~648 ms ~652 ms ~660 ms | |
#245 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~648 ms ~652 ms ~660 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 566 tok/s 509 tok/s 417 tok/s | ~37 ms ~37 ms ~38 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 513 tok/s 439 tok/s 333 tok/s | ~60 ms ~61 ms ~61 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 513 tok/s 439 tok/s 333 tok/s | ~60 ms ~61 ms ~61 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 503 tok/s 462 tok/s 385 tok/s | ~37 ms ~38 ms ~38 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 503 tok/s 462 tok/s 385 tok/s | ~37 ms ~38 ms ~38 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 489 tok/s 411 tok/s 303 tok/s | ~72 ms ~72 ms ~73 ms | |
- | PixArt-α XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 487 tok/s 408 tok/s 300 tok/s | ~73 ms ~74 ms ~74 ms | |
- | PixArt-Σ XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 487 tok/s 408 tok/s 300 tok/s | ~73 ms ~74 ms ~74 ms | |
- | Nemotron 3.5 Parse | 0.63B | INT4 Q8 FP16 | 482 tok/s 402 tok/s 294 tok/s | ~76 ms ~76 ms ~77 ms | |
- | SD 2.1 Turbo | 0.86B | INT4 Q8 FP16 | 435 tok/s 350 tok/s 244 tok/s | ~104 ms ~105 ms ~106 ms | |
- | Stable Diffusion 1.4 | 0.86B | INT4 Q8 FP16 | 435 tok/s 350 tok/s 244 tok/s | ~104 ms ~105 ms ~106 ms | |
- | Stable Diffusion 2.1 | 0.86B | INT4 Q8 FP16 | 435 tok/s 350 tok/s 244 tok/s | ~104 ms ~105 ms ~106 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 430 tok/s 335 tok/s 226 tok/s | ~122 ms ~123 ms ~124 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 409 tok/s 323 tok/s 220 tok/s | ~122 ms ~123 ms ~124 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 409 tok/s 323 tok/s 220 tok/s | ~122 ms ~123 ms ~124 ms | |
- | Kandinsky 2.2 Decoder | 1.22B | INT4 Q8 FP16 | 379 tok/s 291 tok/s 192 tok/s | ~147 ms ~148 ms ~149 ms | |
- | Wan 2.1 T2V 1.3B | 1.3B | INT4 Q8 FP16 | 369 tok/s 282 tok/s 184 tok/s | ~155 ms ~156 ms ~158 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 346 tok/s 259 tok/s 166 tok/s | ~178 ms ~179 ms ~181 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 346 tok/s 259 tok/s 166 tok/s | ~178 ms ~179 ms ~181 ms | |
- | SVD Image-to-Video XT | 1.5B | INT4 Q8 FP16 | 346 tok/s 259 tok/s 166 tok/s | ~178 ms ~179 ms ~181 ms | |
- | SVD Image-to-Video XT 1.1 | 1.5B | INT4 Q8 FP16 | 346 tok/s 259 tok/s 166 tok/s | ~178 ms ~179 ms ~181 ms | |
- | LFM3 1.7B | 1.7B | INT4 Q8 FP16 | 326 tok/s 240 tok/s 151 tok/s | ~201 ms ~202 ms ~204 ms | |
- | LFM3 1.7B Base | 1.7B | INT4 Q8 FP16 | 326 tok/s 240 tok/s 151 tok/s | ~201 ms ~202 ms ~204 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 326 tok/s 240 tok/s 151 tok/s | ~201 ms ~202 ms ~204 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 317 tok/s 231 tok/s 144 tok/s | ~211 ms ~212 ms ~215 ms | |
- | CogVideoX 2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
- | LFM3 Audio 2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
- | LFM3 Omni 2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
- | LFM3 VL 2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
- | Liquid D1 2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
- | LTX-Video 0.9.1 | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
- | Moondream 3.1 | 9B (2B active) | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~264 ms ~265 ms ~268 ms | |
- | Stable Diffusion 3 Medium | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
- | Kandinsky 3.0 | 2.1B | INT4 Q8 FP16 | 293 tok/s 209 tok/s 128 tok/s | ~244 ms ~246 ms ~249 ms | |
- | LTX-2 | 2.2B | INT4 Q8 FP16 | 286 tok/s 203 tok/s 123 tok/s | ~255 ms ~256 ms ~259 ms | |
- | SDXL 1.0 Refiner | 2.31B | INT4 Q8 FP16 | 279 tok/s 196 tok/s 119 tok/s | ~266 ms ~267 ms ~271 ms | |
- | Stable Diffusion 3.5 Medium | 2.5B | INT4 Q8 FP16 | 267 tok/s 186 tok/s 111 tok/s | ~285 ms ~287 ms ~291 ms | |
- | SDXL 1.0 Base | 2.56B | INT4 Q8 FP16 | 264 tok/s 183 tok/s 109 tok/s | ~291 ms ~293 ms ~297 ms | |
- | SDXL Turbo | 2.56B | INT4 Q8 FP16 | 264 tok/s 183 tok/s 109 tok/s | ~291 ms ~293 ms ~297 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 255 tok/s 169 tok/s 98 tok/s | ~336 ms ~338 ms ~342 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 250 tok/s 211 tok/s 151 tok/s | ~156 ms ~157 ms ~159 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 250 tok/s 211 tok/s 151 tok/s | ~156 ms ~157 ms ~159 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 250 tok/s 211 tok/s 151 tok/s | ~156 ms ~157 ms ~159 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 Q8 FP16 | 245 tok/s 176 tok/s 107 tok/s | ~329 ms ~330 ms ~334 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 244 tok/s 165 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | LFM2.5 24B A2B | 24B (2.5B active) | INT4 Q8 FP16 | 225 tok/s 166 tok/s 93 tok/s | ~372 ms ~374 ms ~443 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 Q8 FP16 | 211 tok/s 150 tok/s 86 tok/s | ~411 ms ~413 ms ~453 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 Q8 FP16 | 211 tok/s 150 tok/s 86 tok/s | ~411 ms ~413 ms ~453 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 Q8 | 204 tok/s 147 tok/s | ~432 ms ~434 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 Q8 | 204 tok/s 147 tok/s | ~432 ms ~434 ms | |
- | FLUX.2 Klein 4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
- | Hy-Memory 30B A3B | 30B (3B active) | INT4 Q8 | 201 tok/s 146 tok/s | ~439 ms ~441 ms | |
- | Hy-MT2 30B A3B | 30B (3B active) | INT4 Q8 | 201 tok/s 146 tok/s | ~439 ms ~441 ms | |
- | Mage-VL Cowork | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
- | MiniCPM-V 4.7 35B A3B | 35B (3B active) | INT4 Q8 | 199 tok/s 144 tok/s | ~448 ms ~450 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 193 tok/s 158 tok/s 108 tok/s | ~236 ms ~237 ms ~240 ms | |
- | CogVideoX 1.5 5B | 5B | INT4 Q8 FP16 | 173 tok/s 110 tok/s 61 tok/s | ~542 ms ~546 ms ~553 ms | |
- | CogVideoX 5B | 5B | INT4 Q8 FP16 | 173 tok/s 110 tok/s 61 tok/s | ~542 ms ~546 ms ~553 ms | |
- | Wan 2.2 TI2V 5B | 5B | INT4 Q8 FP16 | 173 tok/s 110 tok/s 61 tok/s | ~542 ms ~546 ms ~553 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 170 tok/s 134 tok/s 88 tok/s | ~308 ms ~309 ms ~313 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 162 tok/s 125 tok/s 81 tok/s | ~338 ms ~340 ms ~345 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 162 tok/s 125 tok/s 81 tok/s | ~338 ms ~340 ms ~345 ms | |
- | AliceAI Foundation 80B A3B Base | 80B (3B active) | INT4 | 152 tok/s | ~654 ms | |
- | Kolibri 1 | 78B (3.5B active) | INT4 | 144 tok/s | ~702 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 141 tok/s 85 tok/s 46 tok/s | ~748 ms ~752 ms ~763 ms | |
- | AuraFlow v0.2 | 6.8B | INT4 Q8 FP16 | 138 tok/s 85 tok/s 46 tok/s | ~728 ms ~732 ms ~742 ms | |
- | AuraFlow v0.3 | 6.8B | INT4 Q8 FP16 | 138 tok/s 85 tok/s 46 tok/s | ~728 ms ~732 ms ~742 ms | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 130 tok/s 80 tok/s 43 tok/s | ~779 ms ~783 ms ~794 ms | |
- | Youtu Parsing Omni | 8B | INT4 Q8 FP16 | 124 tok/s 75 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
- | Stable Diffusion 3.5 Large | 8.1B | INT4 Q8 FP16 | 120 tok/s 73 tok/s 39 tok/s | ~861 ms ~866 ms ~878 ms | |
- | Stable Diffusion 3.5 Large Turbo | 8.1B | INT4 Q8 FP16 | 120 tok/s 73 tok/s 39 tok/s | ~861 ms ~866 ms ~878 ms | |
- | FLUX.2 Klein 9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~956 ms ~962 ms ~975 ms | |
- | LensVLM 9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~956 ms ~962 ms ~975 ms | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~956 ms ~962 ms ~975 ms | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~648 ms ~652 ms ~660 ms | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~648 ms ~652 ms ~660 ms | |
- | Falcon3-10B | 10B | INT4 Q8 FP16 | 101 tok/s 61 tok/s 32 tok/s | ~1.1s ~1.1s ~1.1s | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 Q8 FP16 | 101 tok/s 88 tok/s 64 tok/s | ~400 ms ~401 ms ~405 ms | |
- | Mochi 1 Preview | 10B | INT4 Q8 FP16 | 101 tok/s 61 tok/s 32 tok/s | ~1.1s ~1.1s ~1.1s | |
- | Falcon2-11B | 11B | INT4 Q8 FP16 | 97 tok/s 57 tok/s 30 tok/s | ~1.2s ~1.2s ~1.2s | |
- | Hunyuan Lite | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~751 ms ~755 ms ~766 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~751 ms ~755 ms ~766 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 FP16 | 95 tok/s 67 tok/s 40 tok/s | ~761 ms ~766 ms ~776 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 FP16 | 95 tok/s 67 tok/s 40 tok/s | ~761 ms ~766 ms ~776 ms | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 FP16 | 88 tok/s 61 tok/s 36 tok/s | ~854 ms ~859 ms ~871 ms | |
- | Typhoon-2-8B | 8B | INT4 Q8 FP16 | 88 tok/s 61 tok/s 36 tok/s | ~854 ms ~859 ms ~871 ms | |
- | FLUX.1 Canny Dev | 11.9B | INT4 Q8 FP16 | 87 tok/s 52 tok/s 27 tok/s | ~1.3s ~1.3s ~1.3s | |
- | FLUX.1 Depth Dev | 11.9B | INT4 Q8 FP16 | 87 tok/s 52 tok/s 27 tok/s | ~1.3s ~1.3s ~1.3s | |
- | FLUX.1 Dev | 11.9B | INT4 Q8 FP16 | 87 tok/s 52 tok/s 27 tok/s | ~1.3s ~1.3s ~1.3s | |
- | FLUX.1 Fill Dev | 11.9B | INT4 Q8 FP16 | 87 tok/s 52 tok/s 27 tok/s | ~1.3s ~1.3s ~1.3s | |
- | FLUX.1 Schnell | 11.9B | INT4 Q8 FP16 | 87 tok/s 52 tok/s 27 tok/s | ~1.3s ~1.3s ~1.3s | |
- | HunyuanVideo 1.5 720P T2V | 13B | INT4 Q8 FP16 | 81 tok/s 48 tok/s 25 tok/s | ~1.4s ~1.4s ~1.4s | |
- | GLM-4-9B | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~959 ms ~965 ms ~978 ms | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~959 ms ~965 ms ~978 ms | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~959 ms ~965 ms ~978 ms | |
- | Yi-9B | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~959 ms ~965 ms ~978 ms | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 FP16 | 78 tok/s 54 tok/s 31 tok/s | ~979 ms ~985 ms ~998 ms | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 FP16 | 78 tok/s 54 tok/s 31 tok/s | ~979 ms ~985 ms ~998 ms | |
- | Wan 2.1 I2V 14B 720P | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~1.5s ~1.5s ~1.5s | |
- | Wan 2.1 T2V 14B | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~1.5s ~1.5s ~1.5s | |
- | Wan 2.2 T2V A14B | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~1.5s ~1.5s ~1.5s | |
- | FLUX.2 Dev | 15B | INT4 Q8 FP16 | 71 tok/s 42 tok/s 22 tok/s | ~1.6s ~1.6s ~1.6s | |
- | GreenMind-14B-R1 | 14B | INT4 Q8 FP16 | 55 tok/s 37 tok/s 21 tok/s | ~1.5s ~1.5s ~1.5s | |
- | Arete 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
- | AREX-2 | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
- | PPLX Decider v1 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
- | PPLX Decider v1.1 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
- | Agnes 3.0 Flash | 33B | INT4 Q8 | 35 tok/s 20 tok/s | ~3.4s ~3.4s | |
- | Falcon-40B | 40B | INT4 Q8 | 30 tok/s 16 tok/s | ~4.1s ~4.5s | |
- | Kimi-Dev-72B | 72B | INT4 | 12 tok/s | ~8.7s | |
- | Typhoon-2-70B | 70B | INT4 | 12 tok/s | ~8.5s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
32B Models
8-bit
32B Models
16-bit
14B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
32B Models
LoRA
14B Models
Full Parameter Training
7B-8B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA L20 (48GB).
Inference Sweet Spot
Optimized for 14B models at uncompressed or Q8 precision, and 32B models at Q4 with up to 32k context.
Bandwidth & Speed Profile
With 864 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 192 tok/s on an 8B Q4 model and 23 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning