NVIDIA A10G (24GB)
The NVIDIA A10G (24GB) features 24 GB of dedicated VRAM and 600 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
24 GB
Usable Memory Ceiling
24 GB
Bandwidth
600 GB/s
Max Inference Class
32B Models (Q4)
TDP
300W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA A10G (24GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA A10G (24GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#36 | Qwen 3.8 27B | 27B | INT4 | 30 tok/s | ~4.8s | |
#80 | K2 Horizon MoVA 36B A4B | 36B (4B active) | INT4 | 122 tok/s | ~1.0s | |
#86 | Qwen3.5-27B | 27B | INT4 | 30 tok/s | ~4.8s | |
#93 | Sarvam-30B | 32B (2.4B active) | INT4 | 171 tok/s | ~652 ms | |
#99 | Muse Glimmer 30B | 30B | INT4 | 27 tok/s | ~5.3s | |
#102 | Phi-4 Reasoning Plus | 14B | INT4 Q8 | 39 tok/s 26 tok/s | ~2.5s ~2.5s | |
#103 | Qwen3.6 35B A3B | 35B (3B active) | INT4 | 144 tok/s | ~827 ms | |
#105 | MiMo V2 Flash | 15B (309B active) | INT4 Q8 | 3 tok/s 1 tok/s | ~52.2s ~57.0s | |
#110 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 | 138 tok/s | ~855 ms | |
#120 | Qwen3.5-35B-A3B | 35B (3B active) | INT4 | 144 tok/s | ~827 ms | |
#121 | Qwen3.5-9B | 9B | INT4 Q8 FP16 | 81 tok/s 48 tok/s 24 tok/s | ~1.6s ~1.6s ~1.8s | |
#127 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
#130 | Gemma 4 12B | 11.95B | INT4 Q8 | 45 tok/s 30 tok/s | ~2.2s ~2.2s | |
#136 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
#139 | GPT-OSS 20B | 21B (3.6B active) | INT4 | 63 tok/s | ~808 ms | |
#140 | GLM-4V | 9B | INT4 Q8 FP16 | 57 tok/s 39 tok/s 21 tok/s | ~1.6s ~1.6s ~1.8s | |
#145 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 26 tok/s 15 tok/s 8 tok/s | ~5.1s ~5.2s ~5.2s | |
#150 | Qwen3-30B-A3B | 30B (3B active) | INT4 | 154 tok/s | ~746 ms | |
#152 | Granite 4.2 8B | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#157 | Gemma 4 E4B | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#161 | GLM-4 | 32B | INT4 | 18 tok/s | ~6.6s | |
#163 | Gemma 3 27B | 27B | INT4 | 30 tok/s | ~4.8s | |
#164 | Phi-4 | 14B | INT4 Q8 | 54 tok/s 32 tok/s | ~2.5s ~2.5s | |
#166 | Qwen3-32B | 32B | INT4 | 25 tok/s | ~5.6s | |
#168 | DeepSeek-R1 32B | 32B | INT4 | 26 tok/s | ~5.6s | |
#169 | Gemma 4 31B | 30.7B | INT4 | 26 tok/s | ~5.4s | |
#171 | Magistral Small | 24B | INT4 | 25 tok/s | ~4.3s | |
#172 | Qwen2.5-32B | 32B | INT4 | 25 tok/s | ~5.6s | |
#174 | OLMo 3.1 32B Instruct | 32B | INT4 | 18 tok/s | ~6.6s | |
#176 | DeepSeek-R1 14B | 14B | INT4 Q8 | 56 tok/s 32 tok/s | ~2.5s ~2.5s | |
#177 | Ministral 3 14B | 14B | INT4 Q8 | 39 tok/s 26 tok/s | ~2.5s ~2.5s | |
#180 | Gemma 3 12B | 12B | INT4 Q8 FP16 | 63 tok/s 37 tok/s 17 tok/s | ~2.2s ~2.2s ~2.6s | |
#184 | Mistral-Small-2501 | 24B | INT4 | 33 tok/s | ~4.3s | |
#185 | Ministral 3 8B | 8B | INT4 Q8 FP16 | 63 tok/s 44 tok/s 24 tok/s | ~1.5s ~1.5s ~1.6s | |
#186 | Gemma 2 27B | 27B | INT4 | 30 tok/s | ~4.8s | |
#187 | OLMo 3 7B Think | 7B | INT4 Q8 FP16 | 70 tok/s 49 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
#188 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
#189 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
#190 | Qwen2.5-14B | 14B | INT4 Q8 | 54 tok/s 32 tok/s | ~2.5s ~2.5s | |
#191 | OLMo 3.1 32B Think | 32B | INT4 | 18 tok/s | ~6.6s | |
#193 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 128 tok/s 79 tok/s 43 tok/s | ~939 ms ~944 ms ~955 ms | |
#195 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 121 tok/s 92 tok/s 59 tok/s | ~574 ms ~576 ms ~583 ms | |
#196 | DeepSeek-R1 8B | 8B | INT4 Q8 FP16 | 92 tok/s 54 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#198 | Gemma 2 9B | 9B | INT4 Q8 FP16 | 81 tok/s 48 tok/s 24 tok/s | ~1.6s ~1.6s ~1.8s | |
#199 | Nemotron 3.5 Lightning 30B A3B | 30B (3B active) | INT4 | 154 tok/s | ~746 ms | |
#201 | Qwen3-4B | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
#202 | Qwen3-14B | 14B | INT4 Q8 | 54 tok/s 32 tok/s | ~2.5s ~2.5s | |
#203 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
#204 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 102 tok/s 61 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
#205 | OLMo 3 32B Base | 32B | INT4 | 18 tok/s | ~6.6s | |
#206 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 390 tok/s 302 tok/s 201 tok/s | ~162 ms ~163 ms ~165 ms | |
#207 | Llama 3.1 8B | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#207 | Qwen3-8B | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#210 | Nemotron 3.5 Lightning | 30B (3B active) | INT4 | 54 tok/s | ~822 ms | |
#211 | Command R | 35B | INT4 | 16 tok/s | ~7.2s | |
#212 | Llama 3 8B | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#213 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 266 tok/s 187 tok/s 112 tok/s | ~340 ms ~341 ms ~345 ms | |
#214 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 473 tok/s 426 tok/s 344 tok/s | ~55 ms ~55 ms ~56 ms | |
#216 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 437 tok/s 351 tok/s 244 tok/s | ~120 ms ~121 ms ~122 ms | |
#217 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 160 tok/s 102 tok/s 56 tok/s | ~711 ms ~715 ms ~723 ms | |
#218 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
#220 | Ministral-8B-2410 | 8B | INT4 Q8 FP16 | 89 tok/s 53 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
#221 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 286 tok/s 204 tok/s 125 tok/s | ~301 ms ~302 ms ~305 ms | |
#222 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
#223 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 160 tok/s 102 tok/s 56 tok/s | ~711 ms ~715 ms ~723 ms | |
#224 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 121 tok/s 102 tok/s 72 tok/s | ~428 ms ~429 ms ~433 ms | |
#225 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 241 tok/s 166 tok/s 98 tok/s | ~397 ms ~399 ms ~403 ms | |
#226 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 96 tok/s 58 tok/s 31 tok/s | ~1.3s ~1.3s ~1.3s | |
#227 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 350 tok/s 264 tok/s 170 tok/s | ~206 ms ~207 ms ~209 ms | |
#228 | Phi-3-medium | 14B | INT4 Q8 | 54 tok/s 32 tok/s | ~2.5s ~2.5s | |
#229 | Yi-34B | 34B | INT4 | 17 tok/s | ~7.0s | |
#230 | OLMo 3 7B Instruct | 7B | INT4 Q8 FP16 | 70 tok/s 49 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
#231 | Qwen2-7B | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
#232 | Phi-3-small | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
#233 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 96 tok/s 58 tok/s 31 tok/s | ~1.3s ~1.3s ~1.3s | |
#234 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 350 tok/s 264 tok/s 170 tok/s | ~206 ms ~207 ms ~209 ms | |
#236 | Gemma 1 7B | 7B | INT4 Q8 FP16 | 70 tok/s 49 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
#237 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 259 tok/s 174 tok/s 101 tok/s | ~397 ms ~398 ms ~403 ms | |
#238 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 78 tok/s 55 tok/s 33 tok/s | ~1.1s ~1.1s ~1.1s | |
#239 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 78 tok/s 55 tok/s 33 tok/s | ~1.1s ~1.1s ~1.1s | |
#240 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 78 tok/s 55 tok/s 33 tok/s | ~1.1s ~1.1s ~1.1s | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 531 tok/s 462 tok/s 358 tok/s | ~61 ms ~61 ms ~62 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 465 tok/s 382 tok/s 273 tok/s | ~100 ms ~101 ms ~102 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 465 tok/s 382 tok/s 273 tok/s | ~100 ms ~101 ms ~102 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 454 tok/s 407 tok/s 324 tok/s | ~61 ms ~61 ms ~62 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 454 tok/s 407 tok/s 324 tok/s | ~61 ms ~61 ms ~62 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 437 tok/s 351 tok/s 244 tok/s | ~120 ms ~121 ms ~122 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 372 tok/s 275 tok/s 175 tok/s | ~206 ms ~207 ms ~209 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 350 tok/s 264 tok/s 170 tok/s | ~206 ms ~207 ms ~209 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 350 tok/s 264 tok/s 170 tok/s | ~206 ms ~207 ms ~209 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 286 tok/s 204 tok/s 125 tok/s | ~301 ms ~302 ms ~305 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 286 tok/s 204 tok/s 125 tok/s | ~301 ms ~302 ms ~305 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 266 tok/s 187 tok/s 112 tok/s | ~340 ms ~341 ms ~345 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 258 tok/s 180 tok/s 107 tok/s | ~358 ms ~359 ms ~363 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 200 tok/s 128 tok/s 71 tok/s | ~571 ms ~573 ms ~580 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 196 tok/s 162 tok/s 113 tok/s | ~264 ms ~265 ms ~268 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 196 tok/s 162 tok/s 113 tok/s | ~264 ms ~265 ms ~268 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 196 tok/s 162 tok/s 113 tok/s | ~264 ms ~265 ms ~268 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 191 tok/s 124 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 Q8 FP16 | 191 tok/s 133 tok/s 71 tok/s | ~558 ms ~560 ms ~664 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 188 tok/s 123 tok/s 70 tok/s | ~571 ms ~574 ms ~580 ms | |
- | LFM2.5 24B A2B | 24B (2.5B active) | INT4 | 175 tok/s | ~632 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 Q8 | 162 tok/s 101 tok/s | ~698 ms ~824 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 Q8 | 162 tok/s 101 tok/s | ~698 ms ~824 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 | 156 tok/s | ~734 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 | 156 tok/s | ~734 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
- | Hy-Memory 30B A3B | 30B (3B active) | INT4 | 154 tok/s | ~746 ms | |
- | Hy-MT2 30B A3B | 30B (3B active) | INT4 | 154 tok/s | ~746 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 154 tok/s 97 tok/s 54 tok/s | ~746 ms ~750 ms ~758 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 147 tok/s 118 tok/s 79 tok/s | ~399 ms ~401 ms ~405 ms | |
- | MiniCPM-V 4.7 35B A3B | 35B (3B active) | INT4 | 144 tok/s | ~827 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 128 tok/s 99 tok/s 64 tok/s | ~521 ms ~524 ms ~529 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 121 tok/s 92 tok/s 59 tok/s | ~574 ms ~576 ms ~583 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 121 tok/s 92 tok/s 59 tok/s | ~574 ms ~576 ms ~583 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 104 tok/s 62 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 100 tok/s 60 tok/s 32 tok/s | ~1.3s ~1.3s ~1.3s | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 96 tok/s 58 tok/s 31 tok/s | ~1.3s ~1.3s ~1.3s | |
- | Youtu Parsing Omni | 8B | INT4 Q8 FP16 | 92 tok/s 54 tok/s 28 tok/s | ~1.4s ~1.5s ~1.5s | |
- | LensVLM 9B | 9B | INT4 Q8 FP16 | 81 tok/s 48 tok/s 24 tok/s | ~1.6s ~1.6s ~1.8s | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 FP16 | 81 tok/s 48 tok/s 24 tok/s | ~1.6s ~1.6s ~1.8s | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 78 tok/s 55 tok/s 33 tok/s | ~1.1s ~1.1s ~1.1s | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 78 tok/s 55 tok/s 33 tok/s | ~1.1s ~1.1s ~1.1s | |
- | Falcon3-10B | 10B | INT4 Q8 FP16 | 74 tok/s 43 tok/s 22 tok/s | ~1.8s ~1.8s ~2.0s | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 Q8 | 74 tok/s 60 tok/s | ~677 ms ~738 ms | |
- | Falcon2-11B | 11B | INT4 Q8 FP16 | 71 tok/s 41 tok/s 19 tok/s | ~2.0s ~2.0s ~2.4s | |
- | Hunyuan Lite | 7B | INT4 Q8 FP16 | 70 tok/s 49 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
- | OLMo 3 7B Base | 7B | INT4 Q8 FP16 | 70 tok/s 49 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
- | SEA-LION-7B | 7.1B | INT4 Q8 FP16 | 69 tok/s 48 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 FP16 | 69 tok/s 48 tok/s 28 tok/s | ~1.3s ~1.3s ~1.3s | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 FP16 | 63 tok/s 44 tok/s 24 tok/s | ~1.5s ~1.5s ~1.6s | |
- | Typhoon-2-8B | 8B | INT4 Q8 FP16 | 63 tok/s 44 tok/s 24 tok/s | ~1.5s ~1.5s ~1.6s | |
- | GLM-4-9B | 9B | INT4 Q8 FP16 | 57 tok/s 39 tok/s 21 tok/s | ~1.6s ~1.6s ~1.8s | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 FP16 | 57 tok/s 39 tok/s 21 tok/s | ~1.6s ~1.6s ~1.8s | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 FP16 | 57 tok/s 39 tok/s 21 tok/s | ~1.6s ~1.6s ~1.8s | |
- | Yi-9B | 9B | INT4 Q8 FP16 | 57 tok/s 39 tok/s 21 tok/s | ~1.6s ~1.6s ~1.8s | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 FP16 | 56 tok/s 38 tok/s 21 tok/s | ~1.7s ~1.7s ~1.8s | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 FP16 | 56 tok/s 38 tok/s 21 tok/s | ~1.7s ~1.7s ~1.8s | |
- | GreenMind-14B-R1 | 14B | INT4 Q8 | 39 tok/s 26 tok/s | ~2.5s ~2.5s | |
- | AREX-2 | 27B | INT4 | 30 tok/s | ~4.8s | |
- | PPLX Decider v1 27B | 27B | INT4 | 30 tok/s | ~4.8s | |
- | PPLX Decider v1.1 27B | 27B | INT4 | 30 tok/s | ~4.8s | |
- | Agnes 3.0 Flash | 33B | INT4 | 25 tok/s | ~5.8s | |
- | Falcon-40B | 40B | INT4 | 20 tok/s | ~7.6s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
32B Models
8-bit
14B Models
16-bit
8B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
32B Models
LoRA
14B Models
Full Parameter Training
3B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA A10G (24GB).
Inference Sweet Spot
Optimized for 14B models at uncompressed or Q8 precision, and 32B models at Q4 with up to 32k context.
Bandwidth & Speed Profile
With 600 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 133 tok/s on an 8B Q4 model and 16 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning