NVIDIA L20 (48GB)
The NVIDIA L20 (48GB) features 48 GB of dedicated VRAM and 864 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
48 GB
Usable Memory Ceiling
48 GB
Bandwidth
864 GB/s
Max Inference Class
32B Models (Q4)
TDP
275W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA L20 (48GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA L20 (48GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#36 | Qwen 3.8 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
#80 | K2 Horizon MoVA 36B A4B | 36B (4B active) | INT4 Q8 | 171 tok/s 119 tok/s | ~550 ms ~553 ms | |
#86 | Qwen3.5-27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
#93 | Sarvam-30B | 32B (2.4B active) | INT4 Q8 | 221 tok/s 166 tok/s | ~384 ms ~385 ms | |
#99 | Muse Glimmer 30B | 30B | INT4 Q8 | 38 tok/s 22 tok/s | ~3.1s ~3.1s | |
#102 | Phi-4 Reasoning Plus | 14B | INT4 Q8 FP16 | 55 tok/s 37 tok/s 21 tok/s | ~1.5s ~1.5s ~1.5s | |
#103 | Qwen3.6 35B A3B | 35B (3B active) | INT4 Q8 | 199 tok/s 144 tok/s | ~448 ms ~450 ms | |
#105 | MiMo V2 Flash | 15B (309B active) | INT4 Q8 FP16 | 4 tok/s 2 tok/s 1 tok/s | ~30.6s ~30.8s ~31.3s | |
#110 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 Q8 | 183 tok/s 126 tok/s | ~503 ms ~506 ms | |
#117 | Nemotron 3.5 Bolt 74B A9B | 74B | INT4 | 15 tok/s | ~8.3s | |
#120 | Qwen3.5-35B-A3B | 35B (3B active) | INT4 Q8 | 199 tok/s 144 tok/s | ~448 ms ~450 ms | |
#121 | Qwen3.5-9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~956 ms ~962 ms ~975 ms | |
#127 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
#130 | Gemma 4 12B | 11.95B | INT4 Q8 FP16 | 63 tok/s 43 tok/s 25 tok/s | ~1.3s ~1.3s ~1.3s | |
#136 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
#139 | GPT-OSS 20B | 21B (3.6B active) | INT4 Q8 FP16 | 88 tok/s 76 tok/s 49 tok/s | ~477 ms ~478 ms ~567 ms | |
#140 | GLM-4V | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~959 ms ~965 ms ~978 ms | |
#145 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 36 tok/s 21 tok/s 11 tok/s | ~3.0s ~3.0s ~3.1s | |
#150 | Qwen3-30B-A3B | 30B (3B active) | INT4 Q8 | 201 tok/s 146 tok/s | ~439 ms ~441 ms | |
#152 | Granite 4.2 8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#157 | Gemma 4 E4B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#158 | Mistral Large 3 | 41B (675B active) | INT4 | 2 tok/s | ~66.9s | |
#159 | Qwen2.5-72B | 72B | INT4 | 16 tok/s | ~8.0s | |
#161 | GLM-4 | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~3.3s ~3.6s | |
#162 | DeepSeek-R1 70B | 70B | INT4 | 18 tok/s | ~7.2s | |
#163 | Gemma 3 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
#164 | Phi-4 | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~1.5s ~1.5s ~1.5s | |
#165 | Llama 3.3 70B | 70B | INT4 | 17 tok/s | ~7.2s | |
#166 | Qwen3-32B | 32B | INT4 Q8 | 36 tok/s 20 tok/s | ~3.3s ~3.3s | |
#167 | Hunyuan Standard | 52B (389B active) | INT4 | 3 tok/s | ~38.6s | |
#168 | DeepSeek-R1 32B | 32B | INT4 Q8 | 37 tok/s 21 tok/s | ~3.3s ~3.3s | |
#169 | Gemma 4 31B | 30.7B | INT4 Q8 | 37 tok/s 21 tok/s | ~3.2s ~3.2s | |
#171 | Magistral Small | 24B | INT4 Q8 | 36 tok/s 23 tok/s | ~2.5s ~2.5s | |
#172 | Qwen2.5-32B | 32B | INT4 Q8 | 36 tok/s 20 tok/s | ~3.3s ~3.3s | |
#174 | OLMo 3.1 32B Instruct | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~3.3s ~3.6s | |
#175 | Llama 3.1 70B | 70B | INT4 | 17 tok/s | ~7.2s | |
#176 | DeepSeek-R1 14B | 14B | INT4 Q8 FP16 | 78 tok/s 45 tok/s 24 tok/s | ~1.5s ~1.5s ~1.5s | |
#177 | Ministral 3 14B | 14B | INT4 Q8 FP16 | 55 tok/s 37 tok/s 21 tok/s | ~1.5s ~1.5s ~1.5s | |
#180 | Gemma 3 12B | 12B | INT4 Q8 FP16 | 87 tok/s 51 tok/s 27 tok/s | ~1.3s ~1.3s ~1.3s | |
#183 | Qwen2-72B | 72B | INT4 | 16 tok/s | ~8.0s | |
#184 | Mistral-Small-2501 | 24B | INT4 Q8 FP16 | 47 tok/s 27 tok/s 13 tok/s | ~2.5s ~2.5s ~3.0s | |
#185 | Ministral 3 8B | 8B | INT4 Q8 FP16 | 88 tok/s 61 tok/s 36 tok/s | ~854 ms ~859 ms ~871 ms | |
#186 | Gemma 2 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
#187 | OLMo 3 7B Think | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~751 ms ~755 ms ~766 ms | |
#188 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
#189 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
#190 | Qwen2.5-14B | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~1.5s ~1.5s ~1.5s | |
#191 | OLMo 3.1 32B Think | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~3.3s ~3.6s | |
#192 | Llama 3 70B | 70B | INT4 | 17 tok/s | ~7.2s | |
#193 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 170 tok/s 109 tok/s 60 tok/s | ~553 ms ~556 ms ~563 ms | |
#195 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 162 tok/s 125 tok/s 81 tok/s | ~338 ms ~340 ms ~345 ms | |
#196 | DeepSeek-R1 8B | 8B | INT4 Q8 FP16 | 124 tok/s 75 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#198 | Gemma 2 9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~956 ms ~962 ms ~975 ms | |
#199 | Nemotron 3.5 Lightning 30B A3B | 30B (3B active) | INT4 Q8 | 201 tok/s 146 tok/s | ~439 ms ~441 ms | |
#200 | Kimi Linear 48B A3B Instruct | 48B (3B active) | INT4 | 67 tok/s | ~484 ms | |
#201 | Qwen3-4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
#202 | Qwen3-14B | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~1.5s ~1.5s ~1.5s | |
#203 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
#204 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 138 tok/s 84 tok/s 46 tok/s | ~748 ms ~753 ms ~763 ms | |
#205 | OLMo 3 32B Base | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~3.3s ~3.6s | |
#206 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 446 tok/s 362 tok/s 255 tok/s | ~97 ms ~97 ms ~98 ms | |
#207 | Llama 3.1 8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#207 | Qwen3-8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#210 | Nemotron 3.5 Lightning | 30B (3B active) | INT4 Q8 | 79 tok/s 71 tok/s | ~447 ms ~448 ms | |
#211 | Command R | 35B | INT4 Q8 | 26 tok/s 16 tok/s | ~3.6s ~4.0s | |
#212 | Llama 3 8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#213 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 326 tok/s 240 tok/s 151 tok/s | ~201 ms ~202 ms ~204 ms | |
#214 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 519 tok/s 479 tok/s 403 tok/s | ~34 ms ~34 ms ~34 ms | |
#216 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 489 tok/s 411 tok/s 303 tok/s | ~72 ms ~72 ms ~73 ms | |
#217 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 208 tok/s 137 tok/s 78 tok/s | ~419 ms ~421 ms ~427 ms | |
#218 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
#219 | Mixtral-8x7B-v0.1 | 46.7B (7B active) | INT4 Q8 | 119 tok/s 70 tok/s | ~869 ms ~1.0s | |
#220 | Ministral-8B-2410 | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
#221 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 346 tok/s 259 tok/s 166 tok/s | ~178 ms ~179 ms ~181 ms | |
#222 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
#223 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 208 tok/s 137 tok/s 78 tok/s | ~419 ms ~421 ms ~427 ms | |
#224 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 162 tok/s 138 tok/s 99 tok/s | ~253 ms ~254 ms ~257 ms | |
#225 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~234 ms ~236 ms ~238 ms | |
#226 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 130 tok/s 80 tok/s 43 tok/s | ~779 ms ~783 ms ~794 ms | |
#227 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 409 tok/s 323 tok/s 220 tok/s | ~122 ms ~123 ms ~124 ms | |
#228 | Phi-3-medium | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~1.5s ~1.5s ~1.5s | |
#229 | Yi-34B | 34B | INT4 Q8 | 27 tok/s 16 tok/s | ~3.5s ~3.8s | |
#230 | OLMo 3 7B Instruct | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~751 ms ~755 ms ~766 ms | |
#231 | Qwen2-7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
#232 | Phi-3-small | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
#233 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 130 tok/s 80 tok/s 43 tok/s | ~779 ms ~783 ms ~794 ms | |
#234 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 409 tok/s 323 tok/s 220 tok/s | ~122 ms ~123 ms ~124 ms | |
#236 | Gemma 1 7B | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~751 ms ~755 ms ~766 ms | |
#237 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 319 tok/s 225 tok/s 136 tok/s | ~234 ms ~235 ms ~238 ms | |
#238 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~648 ms ~652 ms ~660 ms | |
#239 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~648 ms ~652 ms ~660 ms | |
#240 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~648 ms ~652 ms ~660 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 566 tok/s 509 tok/s 417 tok/s | ~37 ms ~37 ms ~38 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 513 tok/s 439 tok/s 333 tok/s | ~60 ms ~61 ms ~61 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 513 tok/s 439 tok/s 333 tok/s | ~60 ms ~61 ms ~61 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 503 tok/s 462 tok/s 385 tok/s | ~37 ms ~38 ms ~38 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 503 tok/s 462 tok/s 385 tok/s | ~37 ms ~38 ms ~38 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 489 tok/s 411 tok/s 303 tok/s | ~72 ms ~72 ms ~73 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 430 tok/s 335 tok/s 226 tok/s | ~122 ms ~123 ms ~124 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 409 tok/s 323 tok/s 220 tok/s | ~122 ms ~123 ms ~124 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 409 tok/s 323 tok/s 220 tok/s | ~122 ms ~123 ms ~124 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 346 tok/s 259 tok/s 166 tok/s | ~178 ms ~179 ms ~181 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 346 tok/s 259 tok/s 166 tok/s | ~178 ms ~179 ms ~181 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 326 tok/s 240 tok/s 151 tok/s | ~201 ms ~202 ms ~204 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 317 tok/s 231 tok/s 144 tok/s | ~211 ms ~212 ms ~215 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 255 tok/s 169 tok/s 98 tok/s | ~336 ms ~338 ms ~342 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 250 tok/s 211 tok/s 151 tok/s | ~156 ms ~157 ms ~159 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 250 tok/s 211 tok/s 151 tok/s | ~156 ms ~157 ms ~159 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 250 tok/s 211 tok/s 151 tok/s | ~156 ms ~157 ms ~159 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 Q8 FP16 | 245 tok/s 176 tok/s 107 tok/s | ~329 ms ~330 ms ~334 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 244 tok/s 165 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~336 ms ~338 ms ~343 ms | |
- | LFM2.5 24B A2B | 24B (2.5B active) | INT4 Q8 FP16 | 225 tok/s 166 tok/s 93 tok/s | ~372 ms ~374 ms ~443 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 Q8 FP16 | 211 tok/s 150 tok/s 86 tok/s | ~411 ms ~413 ms ~453 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 Q8 FP16 | 211 tok/s 150 tok/s 86 tok/s | ~411 ms ~413 ms ~453 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 Q8 | 204 tok/s 147 tok/s | ~432 ms ~434 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 Q8 | 204 tok/s 147 tok/s | ~432 ms ~434 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
- | Hy-Memory 30B A3B | 30B (3B active) | INT4 Q8 | 201 tok/s 146 tok/s | ~439 ms ~441 ms | |
- | Hy-MT2 30B A3B | 30B (3B active) | INT4 Q8 | 201 tok/s 146 tok/s | ~439 ms ~441 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~439 ms ~442 ms ~448 ms | |
- | MiniCPM-V 4.7 35B A3B | 35B (3B active) | INT4 Q8 | 199 tok/s 144 tok/s | ~448 ms ~450 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 193 tok/s 158 tok/s 108 tok/s | ~236 ms ~237 ms ~240 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 170 tok/s 134 tok/s 88 tok/s | ~308 ms ~309 ms ~313 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 162 tok/s 125 tok/s 81 tok/s | ~338 ms ~340 ms ~345 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 162 tok/s 125 tok/s 81 tok/s | ~338 ms ~340 ms ~345 ms | |
- | AliceAI Foundation 80B A3B Base | 80B (3B active) | INT4 | 152 tok/s | ~654 ms | |
- | Kolibri 1 | 78B (3.5B active) | INT4 | 144 tok/s | ~702 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 141 tok/s 85 tok/s 46 tok/s | ~748 ms ~752 ms ~763 ms | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~748 ms ~753 ms ~763 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 130 tok/s 80 tok/s 43 tok/s | ~779 ms ~783 ms ~794 ms | |
- | Youtu Parsing Omni | 8B | INT4 Q8 FP16 | 124 tok/s 75 tok/s 40 tok/s | ~851 ms ~856 ms ~868 ms | |
- | LensVLM 9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~956 ms ~962 ms ~975 ms | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~956 ms ~962 ms ~975 ms | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~648 ms ~652 ms ~660 ms | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~648 ms ~652 ms ~660 ms | |
- | Falcon3-10B | 10B | INT4 Q8 FP16 | 101 tok/s 61 tok/s 32 tok/s | ~1.1s ~1.1s ~1.1s | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 Q8 FP16 | 101 tok/s 88 tok/s 64 tok/s | ~400 ms ~401 ms ~405 ms | |
- | Falcon2-11B | 11B | INT4 Q8 FP16 | 97 tok/s 57 tok/s 30 tok/s | ~1.2s ~1.2s ~1.2s | |
- | Hunyuan Lite | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~751 ms ~755 ms ~766 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~751 ms ~755 ms ~766 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 FP16 | 95 tok/s 67 tok/s 40 tok/s | ~761 ms ~766 ms ~776 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 FP16 | 95 tok/s 67 tok/s 40 tok/s | ~761 ms ~766 ms ~776 ms | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 FP16 | 88 tok/s 61 tok/s 36 tok/s | ~854 ms ~859 ms ~871 ms | |
- | Typhoon-2-8B | 8B | INT4 Q8 FP16 | 88 tok/s 61 tok/s 36 tok/s | ~854 ms ~859 ms ~871 ms | |
- | GLM-4-9B | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~959 ms ~965 ms ~978 ms | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~959 ms ~965 ms ~978 ms | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~959 ms ~965 ms ~978 ms | |
- | Yi-9B | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~959 ms ~965 ms ~978 ms | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 FP16 | 78 tok/s 54 tok/s 31 tok/s | ~979 ms ~985 ms ~998 ms | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 FP16 | 78 tok/s 54 tok/s 31 tok/s | ~979 ms ~985 ms ~998 ms | |
- | GreenMind-14B-R1 | 14B | INT4 Q8 FP16 | 55 tok/s 37 tok/s 21 tok/s | ~1.5s ~1.5s ~1.5s | |
- | AREX-2 | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
- | PPLX Decider v1 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
- | PPLX Decider v1.1 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~2.8s ~2.8s | |
- | Agnes 3.0 Flash | 33B | INT4 Q8 | 35 tok/s 20 tok/s | ~3.4s ~3.4s | |
- | Falcon-40B | 40B | INT4 Q8 | 30 tok/s 16 tok/s | ~4.1s ~4.5s | |
- | Kimi-Dev-72B | 72B | INT4 | 12 tok/s | ~8.7s | |
- | Typhoon-2-70B | 70B | INT4 | 12 tok/s | ~8.5s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
32B Models
8-bit
32B Models
16-bit
14B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
32B Models
LoRA
14B Models
Full Parameter Training
7B-8B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA L20 (48GB).
Inference Sweet Spot
Optimized for 14B models at uncompressed or Q8 precision, and 32B models at Q4 with up to 32k context.
Bandwidth & Speed Profile
With 864 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 192 tok/s on an 8B Q4 model and 23 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning