NVIDIA L4 (24GB)
The NVIDIA L4 (24GB) features 24 GB of dedicated VRAM and 300 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
24 GB
Usable Memory Ceiling
24 GB
Bandwidth
300 GB/s
Max Inference Class
32B Models (Q4)
TDP
72W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA L4 (24GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA L4 (24GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#48 | Qwen 3.8 27B | 27B | INT4 | 15 tok/s | ~2.8s | |
#83 | K2 Horizon MoVA 36B A4B | 36B (4B active) | INT4 | 68 tok/s | ~599 ms | |
#93 | Qwen3.5-27B | 27B | INT4 | 15 tok/s | ~2.8s | |
#105 | Sarvam-30B | 32B (2.4B active) | INT4 | 98 tok/s | ~385 ms | |
#106 | Qwen3.6 35B A3B | 35B (3B active) | INT4 | 81 tok/s | ~487 ms | |
#110 | Muse Glimmer 30B | 30B | INT4 | 14 tok/s | ~3.1s | |
#111 | MiMo V2 Flash | 15B (309B active) | INT4 Q8 | 1 tok/s 1 tok/s | ~30.7s ~34.0s | |
#112 | Phi-4 Reasoning Plus | 14B | INT4 Q8 | 20 tok/s 13 tok/s | ~1.5s ~1.5s | |
#117 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 | 77 tok/s | ~505 ms | |
#125 | Qwen3.5-9B | 9B | INT4 Q8 FP16 | 43 tok/s 25 tok/s 12 tok/s | ~958 ms ~975 ms ~1.1s | |
#126 | Qwen3.5-35B-A3B | 35B (3B active) | INT4 | 81 tok/s | ~487 ms | |
#128 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
#135 | Gemma 4 12B | 11.95B | INT4 Q8 | 23 tok/s 16 tok/s | ~1.3s ~1.3s | |
#140 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
#143 | GPT-OSS 20B | 21B (3.6B active) | INT4 | 33 tok/s | ~489 ms | |
#144 | GLM-4V | 9B | INT4 Q8 FP16 | 30 tok/s 20 tok/s 11 tok/s | ~968 ms ~984 ms ~1.1s | |
#150 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 13 tok/s 8 tok/s 4 tok/s | ~3.0s ~3.1s ~3.2s | |
#153 | Qwen3-30B-A3B | 30B (3B active) | INT4 | 87 tok/s | ~440 ms | |
#158 | Granite 4.2 8B | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#159 | Gemma 4 E4B | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#165 | GLM-4 | 32B | INT4 | 9 tok/s | ~3.9s | |
#166 | Phi-4 | 14B | INT4 Q8 | 28 tok/s 16 tok/s | ~1.5s ~1.5s | |
#167 | Gemma 3 27B | 27B | INT4 | 15 tok/s | ~2.8s | |
#170 | DeepSeek-R1 32B | 32B | INT4 | 13 tok/s | ~3.3s | |
#171 | Qwen3-32B | 32B | INT4 | 13 tok/s | ~3.3s | |
#175 | Magistral Small | 24B | INT4 | 13 tok/s | ~2.5s | |
#176 | OLMo 3.1 32B Instruct | 32B | INT4 | 9 tok/s | ~3.9s | |
#177 | Qwen2.5-32B | 32B | INT4 | 13 tok/s | ~3.3s | |
#178 | OLMo 3 7B Think | 7B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 15 tok/s | ~759 ms ~771 ms ~800 ms | |
#179 | DeepSeek-R1 14B | 14B | INT4 Q8 | 29 tok/s 16 tok/s | ~1.5s ~1.5s | |
#183 | Ministral 3 14B | 14B | INT4 Q8 | 20 tok/s 13 tok/s | ~1.5s ~1.5s | |
#186 | Gemma 3 12B | 12B | INT4 Q8 FP16 | 33 tok/s 19 tok/s 9 tok/s | ~1.3s ~1.3s ~1.6s | |
#188 | Mistral-Small-2501 | 24B | INT4 | 17 tok/s | ~2.5s | |
#189 | Ministral 3 8B | 8B | INT4 Q8 FP16 | 33 tok/s 22 tok/s 12 tok/s | ~862 ms ~877 ms ~986 ms | |
#190 | Gemma 4 31B | 30.7B | INT4 | 13 tok/s | ~3.2s | |
#191 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
#192 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
#193 | Gemma 2 27B | 27B | INT4 | 15 tok/s | ~2.8s | |
#194 | Qwen2.5-14B | 14B | INT4 Q8 | 28 tok/s 16 tok/s | ~1.5s ~1.5s | |
#195 | OLMo 3.1 32B Think | 32B | INT4 | 9 tok/s | ~3.9s | |
#196 | Nemotron 3.5 Lightning | 30B (3B active) | INT4 | 28 tok/s | ~501 ms | |
#200 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 71 tok/s 42 tok/s 22 tok/s | ~554 ms ~564 ms ~585 ms | |
#202 | DeepSeek-R1 8B | 8B | INT4 Q8 FP16 | 49 tok/s 28 tok/s 15 tok/s | ~853 ms ~868 ms ~901 ms | |
#204 | Qwen3-4B | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
#205 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 67 tok/s 50 tok/s 31 tok/s | ~343 ms ~348 ms ~361 ms | |
#206 | OLMo 3 32B Base | 32B | INT4 | 9 tok/s | ~3.9s | |
#207 | Gemma 2 9B | 9B | INT4 Q8 FP16 | 43 tok/s 25 tok/s 12 tok/s | ~958 ms ~975 ms ~1.1s | |
#208 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 275 tok/s 195 tok/s 118 tok/s | ~97 ms ~98 ms ~102 ms | |
#209 | Qwen3-14B | 14B | INT4 Q8 | 28 tok/s 16 tok/s | ~1.5s ~1.5s | |
#210 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
#211 | Qwen3-8B | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#213 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 55 tok/s 32 tok/s 17 tok/s | ~750 ms ~763 ms ~792 ms | |
#214 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 166 tok/s 109 tok/s 61 tok/s | ~201 ms ~204 ms ~211 ms | |
#215 | Llama 3.1 8B | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#216 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 366 tok/s 313 tok/s 232 tok/s | ~34 ms ~35 ms ~36 ms | |
#217 | Command R | 35B | INT4 | 8 tok/s | ~4.3s | |
#218 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 326 tok/s 238 tok/s 149 tok/s | ~72 ms ~73 ms ~76 ms | |
#219 | Llama 3 8B | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#221 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 91 tok/s 55 tok/s 29 tok/s | ~420 ms ~427 ms ~443 ms | |
#223 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
#224 | Ministral-8B-2410 | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#225 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 182 tok/s 120 tok/s 69 tok/s | ~178 ms ~181 ms ~187 ms | |
#226 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
#227 | Nemotron 3.5 Lightning 30B A3B | 30B (3B active) | INT4 | 87 tok/s | ~440 ms | |
#228 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 91 tok/s 55 tok/s 29 tok/s | ~420 ms ~427 ms ~443 ms | |
#229 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 67 tok/s 55 tok/s 38 tok/s | ~259 ms ~262 ms ~270 ms | |
#230 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 52 tok/s 30 tok/s 16 tok/s | ~781 ms ~794 ms ~825 ms | |
#231 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 237 tok/s 164 tok/s 98 tok/s | ~122 ms ~124 ms ~129 ms | |
#232 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
#233 | Phi-3-medium | 14B | INT4 Q8 | 28 tok/s 16 tok/s | ~1.5s ~1.5s | |
#234 | OLMo 3 7B Instruct | 7B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 15 tok/s | ~759 ms ~771 ms ~800 ms | |
#235 | Yi-34B | 34B | INT4 | 9 tok/s | ~4.2s | |
#236 | Qwen2-7B | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
#237 | Phi-3-small | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
#238 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 237 tok/s 164 tok/s 98 tok/s | ~122 ms ~124 ms ~129 ms | |
#239 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 52 tok/s 30 tok/s 16 tok/s | ~781 ms ~794 ms ~825 ms | |
#241 | Gemma 1 7B | 7B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 15 tok/s | ~759 ms ~771 ms ~800 ms | |
#242 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 161 tok/s 100 tok/s 55 tok/s | ~234 ms ~238 ms ~246 ms | |
#243 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 17 tok/s | ~655 ms ~665 ms ~690 ms | |
#244 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 17 tok/s | ~655 ms ~665 ms ~690 ms | |
#245 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 17 tok/s | ~655 ms ~665 ms ~690 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 441 tok/s 353 tok/s 245 tok/s | ~37 ms ~38 ms ~39 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 357 tok/s 268 tok/s 172 tok/s | ~60 ms ~61 ms ~63 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 357 tok/s 268 tok/s 172 tok/s | ~60 ms ~61 ms ~63 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 345 tok/s 293 tok/s 214 tok/s | ~38 ms ~38 ms ~40 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 345 tok/s 293 tok/s 214 tok/s | ~38 ms ~38 ms ~40 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 326 tok/s 238 tok/s 149 tok/s | ~72 ms ~73 ms ~76 ms | |
- | PixArt-α XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 323 tok/s 236 tok/s 147 tok/s | ~73 ms ~74 ms ~77 ms | |
- | PixArt-Σ XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 323 tok/s 236 tok/s 147 tok/s | ~73 ms ~74 ms ~77 ms | |
- | Nemotron 3.5 Parse | 0.63B | INT4 Q8 FP16 | 316 tok/s 230 tok/s 144 tok/s | ~76 ms ~77 ms ~80 ms | |
- | SD 2.1 Turbo | 0.86B | INT4 Q8 FP16 | 263 tok/s 185 tok/s 111 tok/s | ~104 ms ~106 ms ~110 ms | |
- | Stable Diffusion 1.4 | 0.86B | INT4 Q8 FP16 | 263 tok/s 185 tok/s 111 tok/s | ~104 ms ~106 ms ~110 ms | |
- | Stable Diffusion 2.1 | 0.86B | INT4 Q8 FP16 | 263 tok/s 185 tok/s 111 tok/s | ~104 ms ~106 ms ~110 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 258 tok/s 173 tok/s 101 tok/s | ~122 ms ~124 ms ~128 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 237 tok/s 164 tok/s 98 tok/s | ~122 ms ~124 ms ~129 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 237 tok/s 164 tok/s 98 tok/s | ~122 ms ~124 ms ~129 ms | |
- | Kandinsky 2.2 Decoder | 1.22B | INT4 Q8 FP16 | 209 tok/s 142 tok/s 82 tok/s | ~147 ms ~149 ms ~154 ms | |
- | Wan 2.1 T2V 1.3B | 1.3B | INT4 Q8 FP16 | 201 tok/s 135 tok/s 78 tok/s | ~156 ms ~158 ms ~163 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 182 tok/s 120 tok/s 69 tok/s | ~178 ms ~181 ms ~187 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 182 tok/s 120 tok/s 69 tok/s | ~178 ms ~181 ms ~187 ms | |
- | SVD Image-to-Video XT | 1.5B | INT4 Q8 FP16 | 182 tok/s 120 tok/s 69 tok/s | ~178 ms ~181 ms ~187 ms | |
- | SVD Image-to-Video XT 1.1 | 1.5B | INT4 Q8 FP16 | 182 tok/s 120 tok/s 69 tok/s | ~178 ms ~181 ms ~187 ms | |
- | LFM3 1.7B | 1.7B | INT4 Q8 FP16 | 166 tok/s 109 tok/s 61 tok/s | ~201 ms ~204 ms ~211 ms | |
- | LFM3 1.7B Base | 1.7B | INT4 Q8 FP16 | 166 tok/s 109 tok/s 61 tok/s | ~201 ms ~204 ms ~211 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 166 tok/s 109 tok/s 61 tok/s | ~201 ms ~204 ms ~211 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 160 tok/s 104 tok/s 58 tok/s | ~212 ms ~215 ms ~223 ms | |
- | Moondream 3.1 | 9B (2B active) | INT4 Q8 FP16 | 148 tok/s 95 tok/s 50 tok/s | ~264 ms ~268 ms ~299 ms | |
- | CogVideoX 2B | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
- | LFM3 Audio 2B | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
- | LFM3 Omni 2B | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
- | LFM3 VL 2B | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
- | Liquid D1 2B | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
- | LTX-Video 0.9.1 | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
- | Stable Diffusion 3 Medium | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
- | Kandinsky 3.0 | 2.1B | INT4 Q8 FP16 | 142 tok/s 91 tok/s 51 tok/s | ~245 ms ~249 ms ~258 ms | |
- | LTX-2 | 2.2B | INT4 Q8 FP16 | 138 tok/s 88 tok/s 49 tok/s | ~255 ms ~259 ms ~269 ms | |
- | SDXL 1.0 Refiner | 2.31B | INT4 Q8 FP16 | 133 tok/s 84 tok/s 47 tok/s | ~267 ms ~271 ms ~281 ms | |
- | Stable Diffusion 3.5 Medium | 2.5B | INT4 Q8 FP16 | 126 tok/s 79 tok/s 43 tok/s | ~286 ms ~291 ms ~301 ms | |
- | SDXL 1.0 Base | 2.56B | INT4 Q8 FP16 | 123 tok/s 77 tok/s 42 tok/s | ~292 ms ~297 ms ~308 ms | |
- | SDXL Turbo | 2.56B | INT4 Q8 FP16 | 123 tok/s 77 tok/s 42 tok/s | ~292 ms ~297 ms ~308 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 118 tok/s 70 tok/s 37 tok/s | ~337 ms ~342 ms ~355 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 115 tok/s 92 tok/s 62 tok/s | ~159 ms ~161 ms ~167 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 115 tok/s 92 tok/s 62 tok/s | ~159 ms ~161 ms ~167 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 115 tok/s 92 tok/s 62 tok/s | ~159 ms ~161 ms ~167 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 Q8 FP16 | 112 tok/s 74 tok/s 38 tok/s | ~330 ms ~334 ms ~403 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 111 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | LFM2.5 24B A2B | 24B (2.5B active) | INT4 | 100 tok/s | ~373 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 Q8 | 92 tok/s 55 tok/s | ~412 ms ~490 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 Q8 | 92 tok/s 55 tok/s | ~412 ms ~490 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 | 88 tok/s | ~433 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 | 88 tok/s | ~433 ms | |
- | FLUX.2 Klein 4B | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
- | Hy-Memory 30B A3B | 30B (3B active) | INT4 | 87 tok/s | ~440 ms | |
- | Hy-MT2 30B A3B | 30B (3B active) | INT4 | 87 tok/s | ~440 ms | |
- | Mage-VL Cowork | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 83 tok/s 65 tok/s 42 tok/s | ~240 ms ~243 ms ~252 ms | |
- | MiniCPM-V 4.7 35B A3B | 35B (3B active) | INT4 | 81 tok/s | ~487 ms | |
- | CogVideoX 1.5 5B | 5B | INT4 Q8 FP16 | 72 tok/s 43 tok/s 23 tok/s | ~544 ms ~553 ms ~574 ms | |
- | CogVideoX 5B | 5B | INT4 Q8 FP16 | 72 tok/s 43 tok/s 23 tok/s | ~544 ms ~553 ms ~574 ms | |
- | Wan 2.2 TI2V 5B | 5B | INT4 Q8 FP16 | 72 tok/s 43 tok/s 23 tok/s | ~544 ms ~553 ms ~574 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 71 tok/s 53 tok/s 33 tok/s | ~312 ms ~317 ms ~328 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 67 tok/s 50 tok/s 31 tok/s | ~343 ms ~348 ms ~361 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 67 tok/s 50 tok/s 31 tok/s | ~343 ms ~348 ms ~361 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 57 tok/s 32 tok/s 17 tok/s | ~749 ms ~763 ms ~792 ms | |
- | AuraFlow v0.2 | 6.8B | INT4 Q8 FP16 | 55 tok/s 32 tok/s 17 tok/s | ~730 ms ~742 ms ~771 ms | |
- | AuraFlow v0.3 | 6.8B | INT4 Q8 FP16 | 55 tok/s 32 tok/s 17 tok/s | ~730 ms ~742 ms ~771 ms | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 52 tok/s 30 tok/s 16 tok/s | ~781 ms ~794 ms ~825 ms | |
- | Youtu Parsing Omni | 8B | INT4 Q8 FP16 | 49 tok/s 28 tok/s 15 tok/s | ~853 ms ~868 ms ~901 ms | |
- | Stable Diffusion 3.5 Large | 8.1B | INT4 Q8 FP16 | 47 tok/s 27 tok/s 14 tok/s | ~863 ms ~879 ms ~912 ms | |
- | Stable Diffusion 3.5 Large Turbo | 8.1B | INT4 Q8 FP16 | 47 tok/s 27 tok/s 14 tok/s | ~863 ms ~879 ms ~912 ms | |
- | FLUX.2 Klein 9B | 9B | INT4 Q8 FP16 | 43 tok/s 25 tok/s 12 tok/s | ~958 ms ~975 ms ~1.1s | |
- | LensVLM 9B | 9B | INT4 Q8 FP16 | 43 tok/s 25 tok/s 12 tok/s | ~958 ms ~975 ms ~1.1s | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 FP16 | 43 tok/s 25 tok/s 12 tok/s | ~958 ms ~975 ms ~1.1s | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 17 tok/s | ~655 ms ~665 ms ~690 ms | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 17 tok/s | ~655 ms ~665 ms ~690 ms | |
- | Falcon3-10B | 10B | INT4 Q8 FP16 | 39 tok/s 22 tok/s 11 tok/s | ~1.1s ~1.1s ~1.2s | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 Q8 | 39 tok/s 32 tok/s | ~411 ms ~450 ms | |
- | Mochi 1 Preview | 10B | INT4 Q8 FP16 | 39 tok/s 22 tok/s 11 tok/s | ~1.1s ~1.1s ~1.2s | |
- | Falcon2-11B | 11B | INT4 Q8 FP16 | 37 tok/s 21 tok/s 10 tok/s | ~1.2s ~1.2s ~1.4s | |
- | Hunyuan Lite | 7B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 15 tok/s | ~759 ms ~771 ms ~800 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 15 tok/s | ~759 ms ~771 ms ~800 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 14 tok/s | ~769 ms ~782 ms ~811 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 14 tok/s | ~769 ms ~782 ms ~811 ms | |
- | FLUX.1 Canny Dev | 11.9B | INT4 Q8 FP16 | 33 tok/s 19 tok/s 9 tok/s | ~1.3s ~1.3s ~1.6s | |
- | FLUX.1 Depth Dev | 11.9B | INT4 Q8 FP16 | 33 tok/s 19 tok/s 9 tok/s | ~1.3s ~1.3s ~1.6s | |
- | FLUX.1 Dev | 11.9B | INT4 Q8 FP16 | 33 tok/s 19 tok/s 9 tok/s | ~1.3s ~1.3s ~1.6s | |
- | FLUX.1 Fill Dev | 11.9B | INT4 Q8 FP16 | 33 tok/s 19 tok/s 9 tok/s | ~1.3s ~1.3s ~1.6s | |
- | FLUX.1 Schnell | 11.9B | INT4 Q8 FP16 | 33 tok/s 19 tok/s 9 tok/s | ~1.3s ~1.3s ~1.6s | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 FP16 | 33 tok/s 22 tok/s 12 tok/s | ~862 ms ~877 ms ~986 ms | |
- | Typhoon-2-8B | 8B | INT4 Q8 FP16 | 33 tok/s 22 tok/s 12 tok/s | ~862 ms ~877 ms ~986 ms | |
- | GLM-4-9B | 9B | INT4 Q8 FP16 | 30 tok/s 20 tok/s 11 tok/s | ~968 ms ~984 ms ~1.1s | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 FP16 | 30 tok/s 20 tok/s 11 tok/s | ~968 ms ~984 ms ~1.1s | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 FP16 | 30 tok/s 20 tok/s 11 tok/s | ~968 ms ~984 ms ~1.1s | |
- | HunyuanVideo 1.5 720P T2V | 13B | INT4 Q8 | 30 tok/s 17 tok/s | ~1.4s ~1.4s | |
- | Yi-9B | 9B | INT4 Q8 FP16 | 30 tok/s 20 tok/s 11 tok/s | ~968 ms ~984 ms ~1.1s | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 FP16 | 29 tok/s 20 tok/s 11 tok/s | ~989 ms ~1.0s ~1.1s | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 FP16 | 29 tok/s 20 tok/s 11 tok/s | ~989 ms ~1.0s ~1.1s | |
- | Wan 2.1 I2V 14B 720P | 14B | INT4 Q8 | 28 tok/s 16 tok/s | ~1.5s ~1.5s | |
- | Wan 2.1 T2V 14B | 14B | INT4 Q8 | 28 tok/s 16 tok/s | ~1.5s ~1.5s | |
- | Wan 2.2 T2V A14B | 14B | INT4 Q8 | 28 tok/s 16 tok/s | ~1.5s ~1.5s | |
- | FLUX.2 Dev | 15B | INT4 Q8 | 27 tok/s 15 tok/s | ~1.6s ~1.6s | |
- | GreenMind-14B-R1 | 14B | INT4 Q8 | 20 tok/s 13 tok/s | ~1.5s ~1.5s | |
- | Arete 27B | 27B | INT4 | 15 tok/s | ~2.8s | |
- | AREX-2 | 27B | INT4 | 15 tok/s | ~2.8s | |
- | PPLX Decider v1 27B | 27B | INT4 | 15 tok/s | ~2.8s | |
- | PPLX Decider v1.1 27B | 27B | INT4 | 15 tok/s | ~2.8s | |
- | Agnes 3.0 Flash | 33B | INT4 | 13 tok/s | ~3.4s | |
- | Falcon-40B | 40B | INT4 | 10 tok/s | ~4.5s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
32B Models
8-bit
14B Models
16-bit
8B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
32B Models
LoRA
14B Models
Full Parameter Training
3B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA L4 (24GB).
Inference Sweet Spot
Optimized for 14B models at uncompressed or Q8 precision, and 32B models at Q4 with up to 32k context.
Bandwidth & Speed Profile
With 300 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 67 tok/s on an 8B Q4 model and 8 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning