NVIDIA L4 (24GB)
The NVIDIA L4 (24GB) features 24 GB of dedicated VRAM and 300 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
24 GB
Usable Memory Ceiling
24 GB
Bandwidth
300 GB/s
Max Inference Class
32B Models (Q4)
TDP
72W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA L4 (24GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA L4 (24GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#36 | Qwen 3.8 27B | 27B | INT4 | 15 tok/s | ~2.8s | |
#80 | K2 Horizon MoVA 36B A4B | 36B (4B active) | INT4 | 68 tok/s | ~599 ms | |
#86 | Qwen3.5-27B | 27B | INT4 | 15 tok/s | ~2.8s | |
#93 | Sarvam-30B | 32B (2.4B active) | INT4 | 98 tok/s | ~385 ms | |
#99 | Muse Glimmer 30B | 30B | INT4 | 14 tok/s | ~3.1s | |
#102 | Phi-4 Reasoning Plus | 14B | INT4 Q8 | 20 tok/s 13 tok/s | ~1.5s ~1.5s | |
#103 | Qwen3.6 35B A3B | 35B (3B active) | INT4 | 81 tok/s | ~487 ms | |
#105 | MiMo V2 Flash | 15B (309B active) | INT4 Q8 | 1 tok/s 1 tok/s | ~30.7s ~34.0s | |
#110 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 | 77 tok/s | ~505 ms | |
#120 | Qwen3.5-35B-A3B | 35B (3B active) | INT4 | 81 tok/s | ~487 ms | |
#121 | Qwen3.5-9B | 9B | INT4 Q8 FP16 | 43 tok/s 25 tok/s 12 tok/s | ~958 ms ~975 ms ~1.1s | |
#127 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
#130 | Gemma 4 12B | 11.95B | INT4 Q8 | 23 tok/s 16 tok/s | ~1.3s ~1.3s | |
#136 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
#139 | GPT-OSS 20B | 21B (3.6B active) | INT4 | 33 tok/s | ~489 ms | |
#140 | GLM-4V | 9B | INT4 Q8 FP16 | 30 tok/s 20 tok/s 11 tok/s | ~968 ms ~984 ms ~1.1s | |
#145 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 13 tok/s 8 tok/s 4 tok/s | ~3.0s ~3.1s ~3.2s | |
#150 | Qwen3-30B-A3B | 30B (3B active) | INT4 | 87 tok/s | ~440 ms | |
#152 | Granite 4.2 8B | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#157 | Gemma 4 E4B | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#161 | GLM-4 | 32B | INT4 | 9 tok/s | ~3.9s | |
#163 | Gemma 3 27B | 27B | INT4 | 15 tok/s | ~2.8s | |
#164 | Phi-4 | 14B | INT4 Q8 | 28 tok/s 16 tok/s | ~1.5s ~1.5s | |
#166 | Qwen3-32B | 32B | INT4 | 13 tok/s | ~3.3s | |
#168 | DeepSeek-R1 32B | 32B | INT4 | 13 tok/s | ~3.3s | |
#169 | Gemma 4 31B | 30.7B | INT4 | 13 tok/s | ~3.2s | |
#171 | Magistral Small | 24B | INT4 | 13 tok/s | ~2.5s | |
#172 | Qwen2.5-32B | 32B | INT4 | 13 tok/s | ~3.3s | |
#174 | OLMo 3.1 32B Instruct | 32B | INT4 | 9 tok/s | ~3.9s | |
#176 | DeepSeek-R1 14B | 14B | INT4 Q8 | 29 tok/s 16 tok/s | ~1.5s ~1.5s | |
#177 | Ministral 3 14B | 14B | INT4 Q8 | 20 tok/s 13 tok/s | ~1.5s ~1.5s | |
#180 | Gemma 3 12B | 12B | INT4 Q8 FP16 | 33 tok/s 19 tok/s 9 tok/s | ~1.3s ~1.3s ~1.6s | |
#184 | Mistral-Small-2501 | 24B | INT4 | 17 tok/s | ~2.5s | |
#185 | Ministral 3 8B | 8B | INT4 Q8 FP16 | 33 tok/s 22 tok/s 12 tok/s | ~862 ms ~877 ms ~986 ms | |
#186 | Gemma 2 27B | 27B | INT4 | 15 tok/s | ~2.8s | |
#187 | OLMo 3 7B Think | 7B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 15 tok/s | ~759 ms ~771 ms ~800 ms | |
#188 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
#189 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
#190 | Qwen2.5-14B | 14B | INT4 Q8 | 28 tok/s 16 tok/s | ~1.5s ~1.5s | |
#191 | OLMo 3.1 32B Think | 32B | INT4 | 9 tok/s | ~3.9s | |
#193 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 71 tok/s 42 tok/s 22 tok/s | ~554 ms ~564 ms ~585 ms | |
#195 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 67 tok/s 50 tok/s 31 tok/s | ~343 ms ~348 ms ~361 ms | |
#196 | DeepSeek-R1 8B | 8B | INT4 Q8 FP16 | 49 tok/s 28 tok/s 15 tok/s | ~853 ms ~868 ms ~901 ms | |
#198 | Gemma 2 9B | 9B | INT4 Q8 FP16 | 43 tok/s 25 tok/s 12 tok/s | ~958 ms ~975 ms ~1.1s | |
#199 | Nemotron 3.5 Lightning 30B A3B | 30B (3B active) | INT4 | 87 tok/s | ~440 ms | |
#201 | Qwen3-4B | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
#202 | Qwen3-14B | 14B | INT4 Q8 | 28 tok/s 16 tok/s | ~1.5s ~1.5s | |
#203 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
#204 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 55 tok/s 32 tok/s 17 tok/s | ~750 ms ~763 ms ~792 ms | |
#205 | OLMo 3 32B Base | 32B | INT4 | 9 tok/s | ~3.9s | |
#206 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 275 tok/s 195 tok/s 118 tok/s | ~97 ms ~98 ms ~102 ms | |
#207 | Llama 3.1 8B | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#207 | Qwen3-8B | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#210 | Nemotron 3.5 Lightning | 30B (3B active) | INT4 | 28 tok/s | ~501 ms | |
#211 | Command R | 35B | INT4 | 8 tok/s | ~4.3s | |
#212 | Llama 3 8B | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#213 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 166 tok/s 109 tok/s 61 tok/s | ~201 ms ~204 ms ~211 ms | |
#214 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 366 tok/s 313 tok/s 232 tok/s | ~34 ms ~35 ms ~36 ms | |
#216 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 326 tok/s 238 tok/s 149 tok/s | ~72 ms ~73 ms ~76 ms | |
#217 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 91 tok/s 55 tok/s 29 tok/s | ~420 ms ~427 ms ~443 ms | |
#218 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
#220 | Ministral-8B-2410 | 8B | INT4 Q8 FP16 | 48 tok/s 28 tok/s 14 tok/s | ~853 ms ~868 ms ~902 ms | |
#221 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 182 tok/s 120 tok/s 69 tok/s | ~178 ms ~181 ms ~187 ms | |
#222 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
#223 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 91 tok/s 55 tok/s 29 tok/s | ~420 ms ~427 ms ~443 ms | |
#224 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 67 tok/s 55 tok/s 38 tok/s | ~259 ms ~262 ms ~270 ms | |
#225 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 147 tok/s 95 tok/s 53 tok/s | ~235 ms ~239 ms ~247 ms | |
#226 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 52 tok/s 30 tok/s 16 tok/s | ~781 ms ~794 ms ~825 ms | |
#227 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 237 tok/s 164 tok/s 98 tok/s | ~122 ms ~124 ms ~129 ms | |
#228 | Phi-3-medium | 14B | INT4 Q8 | 28 tok/s 16 tok/s | ~1.5s ~1.5s | |
#229 | Yi-34B | 34B | INT4 | 9 tok/s | ~4.2s | |
#230 | OLMo 3 7B Instruct | 7B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 15 tok/s | ~759 ms ~771 ms ~800 ms | |
#231 | Qwen2-7B | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
#232 | Phi-3-small | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
#233 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 52 tok/s 30 tok/s 16 tok/s | ~781 ms ~794 ms ~825 ms | |
#234 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 237 tok/s 164 tok/s 98 tok/s | ~122 ms ~124 ms ~129 ms | |
#236 | Gemma 1 7B | 7B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 15 tok/s | ~759 ms ~771 ms ~800 ms | |
#237 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 161 tok/s 100 tok/s 55 tok/s | ~234 ms ~238 ms ~246 ms | |
#238 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 17 tok/s | ~655 ms ~665 ms ~690 ms | |
#239 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 17 tok/s | ~655 ms ~665 ms ~690 ms | |
#240 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 17 tok/s | ~655 ms ~665 ms ~690 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 441 tok/s 353 tok/s 245 tok/s | ~37 ms ~38 ms ~39 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 357 tok/s 268 tok/s 172 tok/s | ~60 ms ~61 ms ~63 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 357 tok/s 268 tok/s 172 tok/s | ~60 ms ~61 ms ~63 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 345 tok/s 293 tok/s 214 tok/s | ~38 ms ~38 ms ~40 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 345 tok/s 293 tok/s 214 tok/s | ~38 ms ~38 ms ~40 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 326 tok/s 238 tok/s 149 tok/s | ~72 ms ~73 ms ~76 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 258 tok/s 173 tok/s 101 tok/s | ~122 ms ~124 ms ~128 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 237 tok/s 164 tok/s 98 tok/s | ~122 ms ~124 ms ~129 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 237 tok/s 164 tok/s 98 tok/s | ~122 ms ~124 ms ~129 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 182 tok/s 120 tok/s 69 tok/s | ~178 ms ~181 ms ~187 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 182 tok/s 120 tok/s 69 tok/s | ~178 ms ~181 ms ~187 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 166 tok/s 109 tok/s 61 tok/s | ~201 ms ~204 ms ~211 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 160 tok/s 104 tok/s 58 tok/s | ~212 ms ~215 ms ~223 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 118 tok/s 70 tok/s 37 tok/s | ~337 ms ~342 ms ~355 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 115 tok/s 92 tok/s 62 tok/s | ~159 ms ~161 ms ~167 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 115 tok/s 92 tok/s 62 tok/s | ~159 ms ~161 ms ~167 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 115 tok/s 92 tok/s 62 tok/s | ~159 ms ~161 ms ~167 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 Q8 FP16 | 112 tok/s 74 tok/s 38 tok/s | ~330 ms ~334 ms ~403 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 111 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 110 tok/s 68 tok/s 37 tok/s | ~337 ms ~343 ms ~355 ms | |
- | LFM2.5 24B A2B | 24B (2.5B active) | INT4 | 100 tok/s | ~373 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 Q8 | 92 tok/s 55 tok/s | ~412 ms ~490 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 Q8 | 92 tok/s 55 tok/s | ~412 ms ~490 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 | 88 tok/s | ~433 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 | 88 tok/s | ~433 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
- | Hy-Memory 30B A3B | 30B (3B active) | INT4 | 87 tok/s | ~440 ms | |
- | Hy-MT2 30B A3B | 30B (3B active) | INT4 | 87 tok/s | ~440 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 87 tok/s 53 tok/s 28 tok/s | ~441 ms ~448 ms ~465 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 83 tok/s 65 tok/s 42 tok/s | ~240 ms ~243 ms ~252 ms | |
- | MiniCPM-V 4.7 35B A3B | 35B (3B active) | INT4 | 81 tok/s | ~487 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 71 tok/s 53 tok/s 33 tok/s | ~312 ms ~317 ms ~328 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 67 tok/s 50 tok/s 31 tok/s | ~343 ms ~348 ms ~361 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 67 tok/s 50 tok/s 31 tok/s | ~343 ms ~348 ms ~361 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 57 tok/s 32 tok/s 17 tok/s | ~749 ms ~763 ms ~792 ms | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 54 tok/s 31 tok/s 16 tok/s | ~750 ms ~763 ms ~793 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 52 tok/s 30 tok/s 16 tok/s | ~781 ms ~794 ms ~825 ms | |
- | Youtu Parsing Omni | 8B | INT4 Q8 FP16 | 49 tok/s 28 tok/s 15 tok/s | ~853 ms ~868 ms ~901 ms | |
- | LensVLM 9B | 9B | INT4 Q8 FP16 | 43 tok/s 25 tok/s 12 tok/s | ~958 ms ~975 ms ~1.1s | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 FP16 | 43 tok/s 25 tok/s 12 tok/s | ~958 ms ~975 ms ~1.1s | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 17 tok/s | ~655 ms ~665 ms ~690 ms | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 42 tok/s 29 tok/s 17 tok/s | ~655 ms ~665 ms ~690 ms | |
- | Falcon3-10B | 10B | INT4 Q8 FP16 | 39 tok/s 22 tok/s 11 tok/s | ~1.1s ~1.1s ~1.2s | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 Q8 | 39 tok/s 32 tok/s | ~411 ms ~450 ms | |
- | Falcon2-11B | 11B | INT4 Q8 FP16 | 37 tok/s 21 tok/s 10 tok/s | ~1.2s ~1.2s ~1.4s | |
- | Hunyuan Lite | 7B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 15 tok/s | ~759 ms ~771 ms ~800 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 15 tok/s | ~759 ms ~771 ms ~800 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 14 tok/s | ~769 ms ~782 ms ~811 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 FP16 | 37 tok/s 25 tok/s 14 tok/s | ~769 ms ~782 ms ~811 ms | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 FP16 | 33 tok/s 22 tok/s 12 tok/s | ~862 ms ~877 ms ~986 ms | |
- | Typhoon-2-8B | 8B | INT4 Q8 FP16 | 33 tok/s 22 tok/s 12 tok/s | ~862 ms ~877 ms ~986 ms | |
- | GLM-4-9B | 9B | INT4 Q8 FP16 | 30 tok/s 20 tok/s 11 tok/s | ~968 ms ~984 ms ~1.1s | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 FP16 | 30 tok/s 20 tok/s 11 tok/s | ~968 ms ~984 ms ~1.1s | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 FP16 | 30 tok/s 20 tok/s 11 tok/s | ~968 ms ~984 ms ~1.1s | |
- | Yi-9B | 9B | INT4 Q8 FP16 | 30 tok/s 20 tok/s 11 tok/s | ~968 ms ~984 ms ~1.1s | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 FP16 | 29 tok/s 20 tok/s 11 tok/s | ~989 ms ~1.0s ~1.1s | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 FP16 | 29 tok/s 20 tok/s 11 tok/s | ~989 ms ~1.0s ~1.1s | |
- | GreenMind-14B-R1 | 14B | INT4 Q8 | 20 tok/s 13 tok/s | ~1.5s ~1.5s | |
- | AREX-2 | 27B | INT4 | 15 tok/s | ~2.8s | |
- | PPLX Decider v1 27B | 27B | INT4 | 15 tok/s | ~2.8s | |
- | PPLX Decider v1.1 27B | 27B | INT4 | 15 tok/s | ~2.8s | |
- | Agnes 3.0 Flash | 33B | INT4 | 13 tok/s | ~3.4s | |
- | Falcon-40B | 40B | INT4 | 10 tok/s | ~4.5s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
32B Models
8-bit
14B Models
16-bit
8B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
32B Models
LoRA
14B Models
Full Parameter Training
3B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA L4 (24GB).
Inference Sweet Spot
Optimized for 14B models at uncompressed or Q8 precision, and 32B models at Q4 with up to 32k context.
Bandwidth & Speed Profile
With 300 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 67 tok/s on an 8B Q4 model and 8 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning