NVIDIA A40 (48GB)
The NVIDIA A40 (48GB) features 48 GB of dedicated VRAM and 696 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
48 GB
Usable Memory Ceiling
48 GB
Bandwidth
696 GB/s
Max Inference Class
32B Models (Q4)
TDP
300W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA A40 (48GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA A40 (48GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#33 | Qwen 3.8 27B | 27B | INT4 Q8 | 34 tok/s 20 tok/s | ~2.2s ~2.3s | |
#48 | Agnes 3.0 Flash | 33B | INT4 Q8 | 28 tok/s 16 tok/s | ~2.7s ~2.8s | |
#72 | K2 Horizon MoVA 36B A4B | 36B (4B active) | INT4 Q8 | 145 tok/s 99 tok/s | ~442 ms ~445 ms | |
#83 | Qwen3.5-27B | 27B | INT4 Q8 | 34 tok/s 20 tok/s | ~2.2s ~2.3s | |
#92 | Sarvam-30B | 32B (2.4B active) | INT4 Q8 | 190 tok/s 141 tok/s | ~309 ms ~310 ms | |
#94 | Qwen3.6 35B A3B | 35B (3B active) | INT4 Q8 | 170 tok/s 121 tok/s | ~360 ms ~362 ms | |
#96 | Muse Glimmer 30B | 30B | INT4 Q8 | 31 tok/s 18 tok/s | ~2.5s ~2.5s | |
#100 | Phi-4 Reasoning Plus | 14B | INT4 Q8 FP16 | 45 tok/s 30 tok/s 17 tok/s | ~1.2s ~1.2s ~1.2s | |
#101 | MiMo V2 Flash | 15B (309B active) | INT4 Q8 FP16 | 3 tok/s 2 tok/s 1 tok/s | ~24.6s ~24.8s ~25.4s | |
#108 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 Q8 | 155 tok/s 106 tok/s | ~404 ms ~407 ms | |
#109 | Gemma 4 12B | 11.95B | INT4 Q8 FP16 | 52 tok/s 35 tok/s 20 tok/s | ~1.0s ~1.0s ~1.0s | |
#113 | Qwen3.5-9B | 9B | INT4 Q8 FP16 | 92 tok/s 55 tok/s 29 tok/s | ~767 ms ~774 ms ~791 ms | |
#114 | Qwen3.5-35B-A3B | 35B (3B active) | INT4 Q8 | 170 tok/s 121 tok/s | ~360 ms ~362 ms | |
#118 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 113 tok/s 69 tok/s 37 tok/s | ~601 ms ~606 ms ~619 ms | |
#128 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 30 tok/s 17 tok/s 9 tok/s | ~2.4s ~2.4s ~2.5s | |
#129 | Gemma 4 31B | 30.7B | INT4 Q8 | 30 tok/s 17 tok/s | ~2.5s ~2.6s | |
#131 | Nemotron 3.5 Lightning | 30B (3B active) | INT4 Q8 | 65 tok/s 59 tok/s | ~362 ms ~364 ms | |
#132 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 172 tok/s 110 tok/s 62 tok/s | ~353 ms ~356 ms ~363 ms | |
#133 | Qwen3-30B-A3B | 30B (3B active) | INT4 Q8 | 172 tok/s 123 tok/s | ~353 ms ~355 ms | |
#136 | GPT-OSS 20B | 21B (3.6B active) | INT4 Q8 FP16 | 72 tok/s 62 tok/s 40 tok/s | ~386 ms ~388 ms ~462 ms | |
#146 | Granite 4.2 8B | 8B | INT4 Q8 FP16 | 101 tok/s 61 tok/s 33 tok/s | ~683 ms ~690 ms ~704 ms | |
#150 | Gemma 4 E4B | 8B | INT4 Q8 FP16 | 101 tok/s 61 tok/s 33 tok/s | ~683 ms ~690 ms ~704 ms | |
#151 | Nemotron 3.5 Lightning 30B A3B | 30B (3B active) | INT4 Q8 | 172 tok/s 123 tok/s | ~353 ms ~355 ms | |
#153 | Qwen3-32B | 32B | INT4 Q8 | 29 tok/s 17 tok/s | ~2.7s ~2.7s | |
#154 | Qwen2.5-72B | 72B | INT4 | 13 tok/s | ~6.4s | |
#155 | Mistral Large 3 | 41B (675B active) | INT4 | 1 tok/s | ~53.6s | |
#156 | GLM-4 | 32B | INT4 Q8 | 23 tok/s 14 tok/s | ~2.7s ~2.9s | |
#157 | DeepSeek-R1 70B | 70B | INT4 | 14 tok/s | ~5.8s | |
#158 | GLM-4V | 9B | INT4 Q8 FP16 | 66 tok/s 45 tok/s 26 tok/s | ~772 ms ~778 ms ~795 ms | |
#159 | Gemma 3 27B | 27B | INT4 Q8 | 34 tok/s 20 tok/s | ~2.2s ~2.3s | |
#160 | Phi-4 | 14B | INT4 Q8 FP16 | 62 tok/s 36 tok/s 19 tok/s | ~1.2s ~1.2s ~1.2s | |
#161 | Llama 3.3 70B | 70B | INT4 | 14 tok/s | ~5.8s | |
#162 | DeepSeek-R1 32B | 32B | INT4 Q8 | 30 tok/s 17 tok/s | ~2.7s ~2.7s | |
#163 | Hunyuan Standard | 52B (389B active) | INT4 | 3 tok/s | ~31.0s | |
#164 | Qwen2.5-32B | 32B | INT4 Q8 | 29 tok/s 17 tok/s | ~2.7s ~2.7s | |
#168 | Magistral Small | 24B | INT4 Q8 | 29 tok/s 19 tok/s | ~2.0s ~2.0s | |
#169 | OLMo 3 7B Think | 7B | INT4 Q8 FP16 | 80 tok/s 56 tok/s 33 tok/s | ~604 ms ~610 ms ~622 ms | |
#170 | OLMo 3.1 32B Instruct | 32B | INT4 Q8 | 23 tok/s 14 tok/s | ~2.7s ~2.9s | |
#171 | Llama 3.1 70B | 70B | INT4 | 14 tok/s | ~5.8s | |
#172 | DeepSeek-R1 14B | 14B | INT4 Q8 FP16 | 64 tok/s 37 tok/s 19 tok/s | ~1.2s ~1.2s ~1.2s | |
#173 | Ministral 3 14B | 14B | INT4 Q8 FP16 | 45 tok/s 30 tok/s 17 tok/s | ~1.2s ~1.2s ~1.2s | |
#177 | Gemma 3 12B | 12B | INT4 Q8 FP16 | 72 tok/s 42 tok/s 22 tok/s | ~1.0s ~1.0s ~1.0s | |
#179 | Qwen2-72B | 72B | INT4 | 13 tok/s | ~6.4s | |
#180 | OLMo 3 32B Base | 32B | INT4 Q8 | 23 tok/s 14 tok/s | ~2.7s ~2.9s | |
#181 | Mistral-Small-2501 | 24B | INT4 Q8 FP16 | 38 tok/s 22 tok/s 10 tok/s | ~2.0s ~2.0s ~2.4s | |
#182 | Ministral 3 8B | 8B | INT4 Q8 FP16 | 72 tok/s 50 tok/s 29 tok/s | ~687 ms ~693 ms ~708 ms | |
#183 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 265 tok/s 185 tok/s 111 tok/s | ~189 ms ~190 ms ~194 ms | |
#184 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 265 tok/s 185 tok/s 111 tok/s | ~189 ms ~190 ms ~194 ms | |
#185 | Qwen2.5-14B | 14B | INT4 Q8 FP16 | 62 tok/s 36 tok/s 19 tok/s | ~1.2s ~1.2s ~1.2s | |
#186 | Gemma 2 27B | 27B | INT4 Q8 | 34 tok/s 20 tok/s | ~2.2s ~2.3s | |
#187 | OLMo 3.1 32B Think | 32B | INT4 Q8 | 23 tok/s 14 tok/s | ~2.7s ~2.9s | |
#188 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 144 tok/s 90 tok/s 49 tok/s | ~444 ms ~448 ms ~457 ms | |
#189 | Qwen3-4B | 4B | INT4 Q8 FP16 | 172 tok/s 110 tok/s 62 tok/s | ~353 ms ~356 ms ~363 ms | |
#190 | DeepSeek-R1 8B | 8B | INT4 Q8 FP16 | 104 tok/s 62 tok/s 33 tok/s | ~683 ms ~689 ms ~704 ms | |
#191 | Llama 3 70B | 70B | INT4 | 14 tok/s | ~5.8s | |
#193 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 137 tok/s 105 tok/s 67 tok/s | ~273 ms ~275 ms ~281 ms | |
#195 | Qwen3-14B | 14B | INT4 Q8 FP16 | 62 tok/s 36 tok/s 19 tok/s | ~1.2s ~1.2s ~1.2s | |
#196 | Kimi Linear 48B A3B Instruct | 48B (3B active) | INT4 | 55 tok/s | ~393 ms | |
#197 | Gemma 2 9B | 9B | INT4 Q8 FP16 | 92 tok/s 55 tok/s 29 tok/s | ~767 ms ~774 ms ~791 ms | |
#198 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 113 tok/s 69 tok/s 37 tok/s | ~601 ms ~606 ms ~619 ms | |
#199 | Qwen3-8B | 8B | INT4 Q8 FP16 | 101 tok/s 61 tok/s 33 tok/s | ~683 ms ~690 ms ~704 ms | |
#200 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 172 tok/s 110 tok/s 62 tok/s | ~353 ms ~356 ms ~363 ms | |
#201 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 116 tok/s 70 tok/s 37 tok/s | ~600 ms ~606 ms ~619 ms | |
#202 | Llama 3.1 8B | 8B | INT4 Q8 FP16 | 101 tok/s 61 tok/s 33 tok/s | ~683 ms ~690 ms ~704 ms | |
#204 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 290 tok/s 208 tok/s 127 tok/s | ~162 ms ~163 ms ~166 ms | |
#205 | Command R | 35B | INT4 Q8 | 21 tok/s 13 tok/s | ~2.9s ~3.2s | |
#206 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 493 tok/s 448 tok/s 368 tok/s | ~28 ms ~28 ms ~29 ms | |
#208 | Llama 3 8B | 8B | INT4 Q8 FP16 | 101 tok/s 61 tok/s 33 tok/s | ~683 ms ~690 ms ~704 ms | |
#209 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 459 tok/s 376 tok/s 267 tok/s | ~59 ms ~59 ms ~60 ms | |
#210 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 179 tok/s 115 tok/s 64 tok/s | ~337 ms ~340 ms ~346 ms | |
#211 | Mixtral-8x7B-v0.1 | 46.7B (7B active) | INT4 Q8 | 100 tok/s 58 tok/s | ~697 ms ~825 ms | |
#212 | Ministral-8B-2410 | 8B | INT4 Q8 FP16 | 101 tok/s 61 tok/s 33 tok/s | ~683 ms ~690 ms ~704 ms | |
#213 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 310 tok/s 226 tok/s 140 tok/s | ~143 ms ~144 ms ~147 ms | |
#214 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 209 tok/s 139 tok/s 79 tok/s | ~270 ms ~273 ms ~278 ms | |
#215 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 413 tok/s 326 tok/s 222 tok/s | ~78 ms ~79 ms ~80 ms | |
#216 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 179 tok/s 115 tok/s 64 tok/s | ~337 ms ~340 ms ~346 ms | |
#217 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 137 tok/s 116 tok/s 82 tok/s | ~205 ms ~206 ms ~209 ms | |
#218 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 109 tok/s 66 tok/s 35 tok/s | ~625 ms ~631 ms ~644 ms | |
#219 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 374 tok/s 288 tok/s 190 tok/s | ~99 ms ~100 ms ~101 ms | |
#220 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 265 tok/s 185 tok/s 111 tok/s | ~189 ms ~190 ms ~194 ms | |
#221 | Phi-3-medium | 14B | INT4 Q8 FP16 | 62 tok/s 36 tok/s 19 tok/s | ~1.2s ~1.2s ~1.2s | |
#222 | OLMo 3 7B Instruct | 7B | INT4 Q8 FP16 | 80 tok/s 56 tok/s 33 tok/s | ~604 ms ~610 ms ~622 ms | |
#223 | Yi-34B | 34B | INT4 Q8 | 22 tok/s 13 tok/s | ~2.8s ~3.1s | |
#224 | Qwen2-7B | 7B | INT4 Q8 FP16 | 113 tok/s 69 tok/s 37 tok/s | ~601 ms ~606 ms ~619 ms | |
#225 | Phi-3-small | 7B | INT4 Q8 FP16 | 113 tok/s 69 tok/s 37 tok/s | ~601 ms ~606 ms ~619 ms | |
#226 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 374 tok/s 288 tok/s 190 tok/s | ~99 ms ~100 ms ~101 ms | |
#227 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 109 tok/s 66 tok/s 35 tok/s | ~625 ms ~631 ms ~644 ms | |
#229 | Gemma 1 7B | 7B | INT4 Q8 FP16 | 80 tok/s 56 tok/s 33 tok/s | ~604 ms ~610 ms ~622 ms | |
#230 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 283 tok/s 194 tok/s 114 tok/s | ~188 ms ~190 ms ~194 ms | |
#231 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 89 tok/s 63 tok/s 38 tok/s | ~522 ms ~526 ms ~537 ms | |
#232 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 89 tok/s 63 tok/s 38 tok/s | ~522 ms ~526 ms ~537 ms | |
#233 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 89 tok/s 63 tok/s 38 tok/s | ~522 ms ~526 ms ~537 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 486 tok/s 406 tok/s 297 tok/s | ~49 ms ~49 ms ~50 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 486 tok/s 406 tok/s 297 tok/s | ~49 ms ~49 ms ~50 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 475 tok/s 430 tok/s 349 tok/s | ~31 ms ~31 ms ~31 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 475 tok/s 430 tok/s 349 tok/s | ~31 ms ~31 ms ~31 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 396 tok/s 299 tok/s 195 tok/s | ~99 ms ~99 ms ~101 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 374 tok/s 288 tok/s 190 tok/s | ~99 ms ~100 ms ~101 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 310 tok/s 226 tok/s 140 tok/s | ~143 ms ~144 ms ~147 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 310 tok/s 226 tok/s 140 tok/s | ~143 ms ~144 ms ~147 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 290 tok/s 208 tok/s 127 tok/s | ~162 ms ~163 ms ~166 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 282 tok/s 200 tok/s 121 tok/s | ~170 ms ~172 ms ~175 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 221 tok/s 144 tok/s 81 tok/s | ~270 ms ~273 ms ~278 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 217 tok/s 181 tok/s 127 tok/s | ~127 ms ~128 ms ~130 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 217 tok/s 181 tok/s 127 tok/s | ~127 ms ~128 ms ~130 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 217 tok/s 181 tok/s 127 tok/s | ~127 ms ~128 ms ~130 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 212 tok/s 140 tok/s 80 tok/s | ~270 ms ~273 ms ~278 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 209 tok/s 139 tok/s 79 tok/s | ~270 ms ~273 ms ~278 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 209 tok/s 139 tok/s 79 tok/s | ~270 ms ~273 ms ~278 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 Q8 FP16 | 181 tok/s 127 tok/s 72 tok/s | ~330 ms ~333 ms ~367 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 Q8 FP16 | 181 tok/s 127 tok/s 72 tok/s | ~330 ms ~333 ms ~367 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 Q8 | 174 tok/s 124 tok/s | ~347 ms ~350 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 Q8 | 174 tok/s 124 tok/s | ~347 ms ~350 ms | |
- | Hy-MT2 30B A3B | 30B (3B active) | INT4 Q8 | 172 tok/s 123 tok/s | ~353 ms ~355 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 172 tok/s 110 tok/s 62 tok/s | ~353 ms ~356 ms ~363 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 144 tok/s 112 tok/s 73 tok/s | ~248 ms ~250 ms ~255 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 137 tok/s 105 tok/s 67 tok/s | ~273 ms ~275 ms ~281 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 137 tok/s 105 tok/s 67 tok/s | ~273 ms ~275 ms ~281 ms | |
- | AliceAI Foundation 80B A3B Base | 80B (3B active) | INT4 | 128 tok/s | ~525 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 118 tok/s 70 tok/s 37 tok/s | ~600 ms ~606 ms ~619 ms | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 113 tok/s 69 tok/s 37 tok/s | ~601 ms ~606 ms ~619 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 113 tok/s 69 tok/s 37 tok/s | ~601 ms ~606 ms ~619 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 109 tok/s 66 tok/s 35 tok/s | ~625 ms ~631 ms ~644 ms | |
- | LensVLM 9B | 9B | INT4 Q8 FP16 | 92 tok/s 55 tok/s 29 tok/s | ~767 ms ~774 ms ~791 ms | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 FP16 | 92 tok/s 55 tok/s 29 tok/s | ~767 ms ~774 ms ~791 ms | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 89 tok/s 63 tok/s 38 tok/s | ~522 ms ~526 ms ~537 ms | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 89 tok/s 63 tok/s 38 tok/s | ~522 ms ~526 ms ~537 ms | |
- | Falcon3-10B | 10B | INT4 Q8 FP16 | 84 tok/s 50 tok/s 26 tok/s | ~850 ms ~858 ms ~876 ms | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 Q8 FP16 | 84 tok/s 73 tok/s 52 tok/s | ~323 ms ~325 ms ~331 ms | |
- | Falcon2-11B | 11B | INT4 Q8 FP16 | 81 tok/s 47 tok/s 24 tok/s | ~932 ms ~941 ms ~961 ms | |
- | Hunyuan Lite | 7B | INT4 Q8 FP16 | 80 tok/s 56 tok/s 33 tok/s | ~604 ms ~610 ms ~622 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 FP16 | 80 tok/s 56 tok/s 33 tok/s | ~604 ms ~610 ms ~622 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 FP16 | 79 tok/s 55 tok/s 32 tok/s | ~612 ms ~618 ms ~631 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 FP16 | 79 tok/s 55 tok/s 32 tok/s | ~612 ms ~618 ms ~631 ms | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 FP16 | 72 tok/s 50 tok/s 29 tok/s | ~687 ms ~693 ms ~708 ms | |
- | Typhoon-2-8B | 8B | INT4 Q8 FP16 | 72 tok/s 50 tok/s 29 tok/s | ~687 ms ~693 ms ~708 ms | |
- | GLM-4-9B | 9B | INT4 Q8 FP16 | 66 tok/s 45 tok/s 26 tok/s | ~772 ms ~778 ms ~795 ms | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 FP16 | 66 tok/s 45 tok/s 26 tok/s | ~772 ms ~778 ms ~795 ms | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 FP16 | 66 tok/s 45 tok/s 26 tok/s | ~772 ms ~778 ms ~795 ms | |
- | Yi-9B | 9B | INT4 Q8 FP16 | 66 tok/s 45 tok/s 26 tok/s | ~772 ms ~778 ms ~795 ms | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 FP16 | 65 tok/s 44 tok/s 26 tok/s | ~788 ms ~795 ms ~811 ms | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 FP16 | 65 tok/s 44 tok/s 26 tok/s | ~788 ms ~795 ms ~811 ms | |
- | GreenMind-14B-R1 | 14B | INT4 Q8 FP16 | 45 tok/s 30 tok/s 17 tok/s | ~1.2s ~1.2s ~1.2s | |
- | Falcon-40B | 40B | INT4 Q8 | 24 tok/s 13 tok/s | ~3.3s ~3.6s | |
- | Kimi-Dev-72B | 72B | INT4 | 10 tok/s | ~7.0s | |
- | Typhoon-2-70B | 70B | INT4 | 10 tok/s | ~6.8s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
32B Models
8-bit
32B Models
16-bit
14B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
32B Models
LoRA
14B Models
Full Parameter Training
7B-8B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA A40 (48GB).
Inference Sweet Spot
Optimized for 14B models at uncompressed or Q8 precision, and 32B models at Q4 with up to 32k context.
Bandwidth & Speed Profile
With 696 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 155 tok/s on an 8B Q4 model and 18 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning