NVIDIA L40S (48GB)
The NVIDIA L40S (48GB) features 48 GB of dedicated VRAM and 864 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
48 GB
Usable Memory Ceiling
48 GB
Bandwidth
864 GB/s
Max Inference Class
32B Models (Q4)
TDP
350W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA L40S (48GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA L40S (48GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#33 | Qwen 3.8 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~944 ms ~962 ms | |
#48 | Agnes 3.0 Flash | 33B | INT4 Q8 | 35 tok/s 20 tok/s | ~1.1s ~1.2s | |
#72 | K2 Horizon MoVA 36B A4B | 36B (4B active) | INT4 Q8 | 171 tok/s 119 tok/s | ~187 ms ~190 ms | |
#83 | Qwen3.5-27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~944 ms ~962 ms | |
#92 | Sarvam-30B | 32B (2.4B active) | INT4 Q8 | 221 tok/s 166 tok/s | ~131 ms ~133 ms | |
#94 | Qwen3.6 35B A3B | 35B (3B active) | INT4 Q8 | 199 tok/s 144 tok/s | ~153 ms ~155 ms | |
#96 | Muse Glimmer 30B | 30B | INT4 Q8 | 38 tok/s 22 tok/s | ~1.0s ~1.1s | |
#100 | Phi-4 Reasoning Plus | 14B | INT4 Q8 FP16 | 55 tok/s 37 tok/s 21 tok/s | ~503 ms ~511 ms ~532 ms | |
#101 | MiMo V2 Flash | 15B (309B active) | INT4 Q8 FP16 | 4 tok/s 2 tok/s 1 tok/s | ~10.3s ~10.5s ~11.0s | |
#108 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 Q8 | 183 tok/s 126 tok/s | ~171 ms ~174 ms | |
#109 | Gemma 4 12B | 11.95B | INT4 Q8 FP16 | 63 tok/s 43 tok/s 25 tok/s | ~431 ms ~438 ms ~456 ms | |
#113 | Qwen3.5-9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~323 ms ~329 ms ~342 ms | |
#114 | Qwen3.5-35B-A3B | 35B (3B active) | INT4 Q8 | 199 tok/s 144 tok/s | ~153 ms ~155 ms | |
#118 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~254 ms ~258 ms ~268 ms | |
#128 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 36 tok/s 21 tok/s 11 tok/s | ~1.0s ~1.0s ~1.1s | |
#129 | Gemma 4 31B | 30.7B | INT4 Q8 | 37 tok/s 21 tok/s | ~1.1s ~1.1s | |
#131 | Nemotron 3.5 Lightning | 30B (3B active) | INT4 Q8 | 79 tok/s 71 tok/s | ~158 ms ~159 ms | |
#132 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~150 ms ~153 ms ~158 ms | |
#133 | Qwen3-30B-A3B | 30B (3B active) | INT4 Q8 | 201 tok/s 146 tok/s | ~150 ms ~152 ms | |
#136 | GPT-OSS 20B | 21B (3.6B active) | INT4 Q8 FP16 | 88 tok/s 76 tok/s 49 tok/s | ~167 ms ~168 ms ~203 ms | |
#146 | Granite 4.2 8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~288 ms ~294 ms ~305 ms | |
#150 | Gemma 4 E4B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~288 ms ~294 ms ~305 ms | |
#151 | Nemotron 3.5 Lightning 30B A3B | 30B (3B active) | INT4 Q8 | 201 tok/s 146 tok/s | ~150 ms ~152 ms | |
#153 | Qwen3-32B | 32B | INT4 Q8 | 36 tok/s 20 tok/s | ~1.1s ~1.1s | |
#154 | Qwen2.5-72B | 72B | INT4 | 16 tok/s | ~2.7s | |
#155 | Mistral Large 3 | 41B (675B active) | INT4 | 2 tok/s | ~22.5s | |
#156 | GLM-4 | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~1.1s ~1.2s | |
#157 | DeepSeek-R1 70B | 70B | INT4 | 18 tok/s | ~2.4s | |
#158 | GLM-4V | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~327 ms ~333 ms ~346 ms | |
#159 | Gemma 3 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~944 ms ~962 ms | |
#160 | Phi-4 | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~498 ms ~507 ms ~527 ms | |
#161 | Llama 3.3 70B | 70B | INT4 | 17 tok/s | ~2.4s | |
#162 | DeepSeek-R1 32B | 32B | INT4 Q8 | 37 tok/s 21 tok/s | ~1.1s ~1.1s | |
#163 | Hunyuan Standard | 52B (389B active) | INT4 | 3 tok/s | ~13.0s | |
#164 | Qwen2.5-32B | 32B | INT4 Q8 | 36 tok/s 20 tok/s | ~1.1s ~1.1s | |
#168 | Magistral Small | 24B | INT4 Q8 | 36 tok/s 23 tok/s | ~848 ms ~863 ms | |
#169 | OLMo 3 7B Think | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~257 ms ~261 ms ~271 ms | |
#170 | OLMo 3.1 32B Instruct | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~1.1s ~1.2s | |
#171 | Llama 3.1 70B | 70B | INT4 | 17 tok/s | ~2.4s | |
#172 | DeepSeek-R1 14B | 14B | INT4 Q8 FP16 | 78 tok/s 45 tok/s 24 tok/s | ~497 ms ~507 ms ~527 ms | |
#173 | Ministral 3 14B | 14B | INT4 Q8 FP16 | 55 tok/s 37 tok/s 21 tok/s | ~503 ms ~511 ms ~532 ms | |
#177 | Gemma 3 12B | 12B | INT4 Q8 FP16 | 87 tok/s 51 tok/s 27 tok/s | ~428 ms ~436 ms ~453 ms | |
#179 | Qwen2-72B | 72B | INT4 | 16 tok/s | ~2.7s | |
#180 | OLMo 3 32B Base | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~1.1s ~1.2s | |
#181 | Mistral-Small-2501 | 24B | INT4 Q8 FP16 | 47 tok/s 27 tok/s 13 tok/s | ~841 ms ~857 ms ~1.0s | |
#182 | Ministral 3 8B | 8B | INT4 Q8 FP16 | 88 tok/s 61 tok/s 36 tok/s | ~291 ms ~296 ms ~308 ms | |
#183 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~81 ms ~82 ms ~85 ms | |
#184 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~81 ms ~82 ms ~85 ms | |
#185 | Qwen2.5-14B | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~498 ms ~507 ms ~527 ms | |
#186 | Gemma 2 27B | 27B | INT4 Q8 | 42 tok/s 24 tok/s | ~944 ms ~962 ms | |
#187 | OLMo 3.1 32B Think | 32B | INT4 Q8 | 28 tok/s 17 tok/s | ~1.1s ~1.2s | |
#188 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 170 tok/s 109 tok/s 60 tok/s | ~188 ms ~191 ms ~199 ms | |
#189 | Qwen3-4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~150 ms ~153 ms ~158 ms | |
#190 | DeepSeek-R1 8B | 8B | INT4 Q8 FP16 | 124 tok/s 75 tok/s 40 tok/s | ~288 ms ~293 ms ~305 ms | |
#191 | Llama 3 70B | 70B | INT4 | 17 tok/s | ~2.4s | |
#193 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 162 tok/s 125 tok/s 81 tok/s | ~117 ms ~119 ms ~124 ms | |
#195 | Qwen3-14B | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~498 ms ~507 ms ~527 ms | |
#196 | Kimi Linear 48B A3B Instruct | 48B (3B active) | INT4 | 67 tok/s | ~171 ms | |
#197 | Gemma 2 9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~323 ms ~329 ms ~342 ms | |
#198 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~254 ms ~258 ms ~268 ms | |
#199 | Qwen3-8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~288 ms ~294 ms ~305 ms | |
#200 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~150 ms ~153 ms ~158 ms | |
#201 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 138 tok/s 84 tok/s 46 tok/s | ~254 ms ~258 ms ~268 ms | |
#202 | Llama 3.1 8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~288 ms ~294 ms ~305 ms | |
#204 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 326 tok/s 240 tok/s 151 tok/s | ~70 ms ~71 ms ~73 ms | |
#205 | Command R | 35B | INT4 Q8 | 26 tok/s 16 tok/s | ~1.2s ~1.4s | |
#206 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 519 tok/s 479 tok/s 403 tok/s | ~14 ms ~14 ms ~14 ms | |
#208 | Llama 3 8B | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~288 ms ~294 ms ~305 ms | |
#209 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 489 tok/s 411 tok/s 303 tok/s | ~26 ms ~27 ms ~28 ms | |
#210 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 208 tok/s 137 tok/s 78 tok/s | ~143 ms ~146 ms ~151 ms | |
#211 | Mixtral-8x7B-v0.1 | 46.7B (7B active) | INT4 Q8 | 119 tok/s 70 tok/s | ~294 ms ~350 ms | |
#212 | Ministral-8B-2410 | 8B | INT4 Q8 FP16 | 121 tok/s 74 tok/s 40 tok/s | ~288 ms ~294 ms ~305 ms | |
#213 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 346 tok/s 259 tok/s 166 tok/s | ~62 ms ~63 ms ~65 ms | |
#214 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~115 ms ~117 ms ~122 ms | |
#215 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 446 tok/s 362 tok/s 255 tok/s | ~35 ms ~35 ms ~36 ms | |
#216 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 208 tok/s 137 tok/s 78 tok/s | ~143 ms ~146 ms ~151 ms | |
#217 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 162 tok/s 138 tok/s 99 tok/s | ~89 ms ~90 ms ~93 ms | |
#218 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 130 tok/s 80 tok/s 43 tok/s | ~264 ms ~269 ms ~279 ms | |
#219 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 409 tok/s 323 tok/s 220 tok/s | ~43 ms ~44 ms ~45 ms | |
#220 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 300 tok/s 216 tok/s 133 tok/s | ~81 ms ~82 ms ~85 ms | |
#221 | Phi-3-medium | 14B | INT4 Q8 FP16 | 76 tok/s 45 tok/s 23 tok/s | ~498 ms ~507 ms ~527 ms | |
#222 | OLMo 3 7B Instruct | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~257 ms ~261 ms ~271 ms | |
#223 | Yi-34B | 34B | INT4 Q8 | 27 tok/s 16 tok/s | ~1.2s ~1.3s | |
#224 | Qwen2-7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~254 ms ~258 ms ~268 ms | |
#225 | Phi-3-small | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~254 ms ~258 ms ~268 ms | |
#226 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 409 tok/s 323 tok/s 220 tok/s | ~43 ms ~44 ms ~45 ms | |
#227 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 130 tok/s 80 tok/s 43 tok/s | ~264 ms ~269 ms ~279 ms | |
#229 | Gemma 1 7B | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~257 ms ~261 ms ~271 ms | |
#230 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 319 tok/s 225 tok/s 136 tok/s | ~81 ms ~82 ms ~85 ms | |
#231 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~222 ms ~226 ms ~234 ms | |
#232 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~222 ms ~226 ms ~234 ms | |
#233 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~222 ms ~226 ms ~234 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 513 tok/s 439 tok/s 333 tok/s | ~23 ms ~23 ms ~24 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 513 tok/s 439 tok/s 333 tok/s | ~23 ms ~23 ms ~24 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 503 tok/s 462 tok/s 385 tok/s | ~15 ms ~15 ms ~16 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 503 tok/s 462 tok/s 385 tok/s | ~15 ms ~15 ms ~16 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 430 tok/s 335 tok/s 226 tok/s | ~43 ms ~44 ms ~45 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 409 tok/s 323 tok/s 220 tok/s | ~43 ms ~44 ms ~45 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 346 tok/s 259 tok/s 166 tok/s | ~62 ms ~63 ms ~65 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 346 tok/s 259 tok/s 166 tok/s | ~62 ms ~63 ms ~65 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 326 tok/s 240 tok/s 151 tok/s | ~70 ms ~71 ms ~73 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 317 tok/s 231 tok/s 144 tok/s | ~73 ms ~74 ms ~77 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 255 tok/s 169 tok/s 98 tok/s | ~115 ms ~117 ms ~121 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 250 tok/s 211 tok/s 151 tok/s | ~56 ms ~56 ms ~58 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 250 tok/s 211 tok/s 151 tok/s | ~56 ms ~56 ms ~58 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 250 tok/s 211 tok/s 151 tok/s | ~56 ms ~56 ms ~58 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 244 tok/s 165 tok/s 96 tok/s | ~115 ms ~117 ms ~122 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~115 ms ~117 ms ~122 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 241 tok/s 164 tok/s 96 tok/s | ~115 ms ~117 ms ~122 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 Q8 FP16 | 211 tok/s 150 tok/s 86 tok/s | ~140 ms ~142 ms ~159 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 Q8 FP16 | 211 tok/s 150 tok/s 86 tok/s | ~140 ms ~142 ms ~159 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 Q8 | 204 tok/s 147 tok/s | ~148 ms ~149 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 Q8 | 204 tok/s 147 tok/s | ~148 ms ~149 ms | |
- | Hy-MT2 30B A3B | 30B (3B active) | INT4 Q8 | 201 tok/s 146 tok/s | ~150 ms ~152 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 201 tok/s 132 tok/s 75 tok/s | ~150 ms ~153 ms ~158 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 170 tok/s 134 tok/s 88 tok/s | ~107 ms ~109 ms ~113 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 162 tok/s 125 tok/s 81 tok/s | ~117 ms ~119 ms ~124 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 162 tok/s 125 tok/s 81 tok/s | ~117 ms ~119 ms ~124 ms | |
- | AliceAI Foundation 80B A3B Base | 80B (3B active) | INT4 | 152 tok/s | ~222 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 141 tok/s 85 tok/s 46 tok/s | ~253 ms ~258 ms ~268 ms | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~254 ms ~258 ms ~268 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 135 tok/s 83 tok/s 45 tok/s | ~254 ms ~258 ms ~268 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 130 tok/s 80 tok/s 43 tok/s | ~264 ms ~269 ms ~279 ms | |
- | LensVLM 9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~323 ms ~329 ms ~342 ms | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 FP16 | 110 tok/s 67 tok/s 36 tok/s | ~323 ms ~329 ms ~342 ms | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~222 ms ~226 ms ~234 ms | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 107 tok/s 77 tok/s 46 tok/s | ~222 ms ~226 ms ~234 ms | |
- | Falcon3-10B | 10B | INT4 Q8 FP16 | 101 tok/s 61 tok/s 32 tok/s | ~358 ms ~365 ms ~379 ms | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 Q8 FP16 | 101 tok/s 88 tok/s 64 tok/s | ~140 ms ~142 ms ~146 ms | |
- | Falcon2-11B | 11B | INT4 Q8 FP16 | 97 tok/s 57 tok/s 30 tok/s | ~392 ms ~400 ms ~416 ms | |
- | Hunyuan Lite | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~257 ms ~261 ms ~271 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 FP16 | 96 tok/s 68 tok/s 40 tok/s | ~257 ms ~261 ms ~271 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 FP16 | 95 tok/s 67 tok/s 40 tok/s | ~260 ms ~265 ms ~275 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 FP16 | 95 tok/s 67 tok/s 40 tok/s | ~260 ms ~265 ms ~275 ms | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 FP16 | 88 tok/s 61 tok/s 36 tok/s | ~291 ms ~296 ms ~308 ms | |
- | Typhoon-2-8B | 8B | INT4 Q8 FP16 | 88 tok/s 61 tok/s 36 tok/s | ~291 ms ~296 ms ~308 ms | |
- | GLM-4-9B | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~327 ms ~333 ms ~346 ms | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~327 ms ~333 ms ~346 ms | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~327 ms ~333 ms ~346 ms | |
- | Yi-9B | 9B | INT4 Q8 FP16 | 80 tok/s 55 tok/s 32 tok/s | ~327 ms ~333 ms ~346 ms | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 FP16 | 78 tok/s 54 tok/s 31 tok/s | ~334 ms ~340 ms ~353 ms | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 FP16 | 78 tok/s 54 tok/s 31 tok/s | ~334 ms ~340 ms ~353 ms | |
- | GreenMind-14B-R1 | 14B | INT4 Q8 FP16 | 55 tok/s 37 tok/s 21 tok/s | ~503 ms ~511 ms ~532 ms | |
- | Falcon-40B | 40B | INT4 Q8 | 30 tok/s 16 tok/s | ~1.4s ~1.5s | |
- | Kimi-Dev-72B | 72B | INT4 | 12 tok/s | ~2.9s | |
- | Typhoon-2-70B | 70B | INT4 | 12 tok/s | ~2.9s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
32B Models
8-bit
32B Models
16-bit
14B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
32B Models
LoRA
14B Models
Full Parameter Training
7B-8B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA L40S (48GB).
Inference Sweet Spot
Optimized for 14B models at uncompressed or Q8 precision, and 32B models at Q4 with up to 32k context.
Bandwidth & Speed Profile
With 864 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 192 tok/s on an 8B Q4 model and 23 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning