NVIDIA A30 (24GB)
The NVIDIA A30 (24GB) features 24 GB of dedicated VRAM and 933 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
24 GB
Usable Memory Ceiling
24 GB
Bandwidth
933 GB/s
Max Inference Class
32B Models (Q4)
TDP
165W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA A30 (24GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA A30 (24GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#36 | Qwen 3.8 27B | 27B | INT4 | 45 tok/s | ~2.0s | |
#80 | K2 Horizon MoVA 36B A4B | 36B (4B active) | INT4 | 172 tok/s | ~435 ms | |
#86 | Qwen3.5-27B | 27B | INT4 | 45 tok/s | ~2.0s | |
#93 | Sarvam-30B | 32B (2.4B active) | INT4 | 232 tok/s | ~280 ms | |
#99 | Muse Glimmer 30B | 30B | INT4 | 41 tok/s | ~2.3s | |
#102 | Phi-4 Reasoning Plus | 14B | INT4 Q8 | 59 tok/s 40 tok/s | ~1.1s ~1.1s | |
#103 | Qwen3.6 35B A3B | 35B (3B active) | INT4 | 199 tok/s | ~354 ms | |
#105 | MiMo V2 Flash | 15B (309B active) | INT4 Q8 | 4 tok/s 2 tok/s | ~22.2s ~24.4s | |
#110 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 | 193 tok/s | ~366 ms | |
#120 | Qwen3.5-35B-A3B | 35B (3B active) | INT4 | 199 tok/s | ~354 ms | |
#121 | Qwen3.5-9B | 9B | INT4 Q8 FP16 | 117 tok/s 72 tok/s 36 tok/s | ~695 ms ~700 ms ~773 ms | |
#127 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 143 tok/s 89 tok/s 49 tok/s | ~544 ms ~548 ms ~558 ms | |
#130 | Gemma 4 12B | 11.95B | INT4 Q8 | 68 tok/s 46 tok/s | ~922 ms ~928 ms | |
#136 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 212 tok/s 140 tok/s 80 tok/s | ~320 ms ~322 ms ~328 ms | |
#139 | GPT-OSS 20B | 21B (3.6B active) | INT4 | 94 tok/s | ~348 ms | |
#140 | GLM-4V | 9B | INT4 Q8 FP16 | 85 tok/s 59 tok/s 33 tok/s | ~698 ms ~703 ms ~776 ms | |
#145 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 39 tok/s 23 tok/s 12 tok/s | ~2.2s ~2.2s ~2.2s | |
#150 | Qwen3-30B-A3B | 30B (3B active) | INT4 | 212 tok/s | ~320 ms | |
#152 | Granite 4.2 8B | 8B | INT4 Q8 FP16 | 129 tok/s 79 tok/s 43 tok/s | ~619 ms ~624 ms ~634 ms | |
#157 | Gemma 4 E4B | 8B | INT4 Q8 FP16 | 129 tok/s 79 tok/s 43 tok/s | ~619 ms ~624 ms ~634 ms | |
#161 | GLM-4 | 32B | INT4 | 27 tok/s | ~2.8s | |
#163 | Gemma 3 27B | 27B | INT4 | 45 tok/s | ~2.0s | |
#164 | Phi-4 | 14B | INT4 Q8 | 81 tok/s 48 tok/s | ~1.1s ~1.1s | |
#166 | Qwen3-32B | 32B | INT4 | 39 tok/s | ~2.4s | |
#168 | DeepSeek-R1 32B | 32B | INT4 | 40 tok/s | ~2.4s | |
#169 | Gemma 4 31B | 30.7B | INT4 | 40 tok/s | ~2.3s | |
#171 | Magistral Small | 24B | INT4 | 38 tok/s | ~1.8s | |
#172 | Qwen2.5-32B | 32B | INT4 | 39 tok/s | ~2.4s | |
#174 | OLMo 3.1 32B Instruct | 32B | INT4 | 27 tok/s | ~2.8s | |
#176 | DeepSeek-R1 14B | 14B | INT4 Q8 | 84 tok/s 49 tok/s | ~1.1s ~1.1s | |
#177 | Ministral 3 14B | 14B | INT4 Q8 | 59 tok/s 40 tok/s | ~1.1s ~1.1s | |
#180 | Gemma 3 12B | 12B | INT4 Q8 FP16 | 92 tok/s 55 tok/s 26 tok/s | ~921 ms ~928 ms ~1.1s | |
#184 | Mistral-Small-2501 | 24B | INT4 | 50 tok/s | ~1.8s | |
#185 | Ministral 3 8B | 8B | INT4 Q8 FP16 | 94 tok/s 65 tok/s 37 tok/s | ~622 ms ~626 ms ~691 ms | |
#186 | Gemma 2 27B | 27B | INT4 | 45 tok/s | ~2.0s | |
#187 | OLMo 3 7B Think | 7B | INT4 Q8 FP16 | 103 tok/s 73 tok/s 43 tok/s | ~547 ms ~551 ms ~560 ms | |
#188 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 312 tok/s 227 tok/s 141 tok/s | ~171 ms ~172 ms ~175 ms | |
#189 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 312 tok/s 227 tok/s 141 tok/s | ~171 ms ~172 ms ~175 ms | |
#190 | Qwen2.5-14B | 14B | INT4 Q8 | 81 tok/s 48 tok/s | ~1.1s ~1.1s | |
#191 | OLMo 3.1 32B Think | 32B | INT4 | 27 tok/s | ~2.8s | |
#193 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 180 tok/s 116 tok/s 65 tok/s | ~402 ms ~405 ms ~412 ms | |
#195 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 171 tok/s 133 tok/s 87 tok/s | ~247 ms ~249 ms ~253 ms | |
#196 | DeepSeek-R1 8B | 8B | INT4 Q8 FP16 | 132 tok/s 80 tok/s 43 tok/s | ~619 ms ~624 ms ~634 ms | |
#198 | Gemma 2 9B | 9B | INT4 Q8 FP16 | 117 tok/s 72 tok/s 36 tok/s | ~695 ms ~700 ms ~773 ms | |
#199 | Nemotron 3.5 Lightning 30B A3B | 30B (3B active) | INT4 | 212 tok/s | ~320 ms | |
#201 | Qwen3-4B | 4B | INT4 Q8 FP16 | 212 tok/s 140 tok/s 80 tok/s | ~320 ms ~322 ms ~328 ms | |
#202 | Qwen3-14B | 14B | INT4 Q8 | 81 tok/s 48 tok/s | ~1.1s ~1.1s | |
#203 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 212 tok/s 140 tok/s 80 tok/s | ~320 ms ~322 ms ~328 ms | |
#204 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 147 tok/s 90 tok/s 49 tok/s | ~544 ms ~548 ms ~558 ms | |
#205 | OLMo 3 32B Base | 32B | INT4 | 27 tok/s | ~2.8s | |
#206 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 457 tok/s 375 tok/s 267 tok/s | ~71 ms ~72 ms ~73 ms | |
#207 | Llama 3.1 8B | 8B | INT4 Q8 FP16 | 129 tok/s 79 tok/s 43 tok/s | ~619 ms ~624 ms ~634 ms | |
#207 | Qwen3-8B | 8B | INT4 Q8 FP16 | 129 tok/s 79 tok/s 43 tok/s | ~619 ms ~624 ms ~634 ms | |
#210 | Nemotron 3.5 Lightning | 30B (3B active) | INT4 | 80 tok/s | ~355 ms | |
#211 | Command R | 35B | INT4 | 25 tok/s | ~3.1s | |
#212 | Llama 3 8B | 8B | INT4 Q8 FP16 | 129 tok/s 79 tok/s 43 tok/s | ~619 ms ~624 ms ~634 ms | |
#213 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 339 tok/s 252 tok/s 160 tok/s | ~147 ms ~148 ms ~150 ms | |
#214 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 528 tok/s 489 tok/s 416 tok/s | ~26 ms ~26 ms ~26 ms | |
#216 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 499 tok/s 423 tok/s 315 tok/s | ~53 ms ~54 ms ~54 ms | |
#217 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 219 tok/s 146 tok/s 84 tok/s | ~305 ms ~307 ms ~312 ms | |
#218 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 143 tok/s 89 tok/s 49 tok/s | ~544 ms ~548 ms ~558 ms | |
#220 | Ministral-8B-2410 | 8B | INT4 Q8 FP16 | 129 tok/s 79 tok/s 43 tok/s | ~619 ms ~624 ms ~634 ms | |
#221 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 359 tok/s 271 tok/s 176 tok/s | ~130 ms ~131 ms ~133 ms | |
#222 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 253 tok/s 173 tok/s 102 tok/s | ~245 ms ~247 ms ~251 ms | |
#223 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 219 tok/s 146 tok/s 84 tok/s | ~305 ms ~307 ms ~312 ms | |
#224 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 171 tok/s 146 tok/s 105 tok/s | ~185 ms ~186 ms ~189 ms | |
#225 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 312 tok/s 227 tok/s 141 tok/s | ~171 ms ~172 ms ~175 ms | |
#226 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 139 tok/s 86 tok/s 47 tok/s | ~566 ms ~571 ms ~580 ms | |
#227 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 421 tok/s 336 tok/s 232 tok/s | ~90 ms ~90 ms ~92 ms | |
#228 | Phi-3-medium | 14B | INT4 Q8 | 81 tok/s 48 tok/s | ~1.1s ~1.1s | |
#229 | Yi-34B | 34B | INT4 | 26 tok/s | ~3.0s | |
#230 | OLMo 3 7B Instruct | 7B | INT4 Q8 FP16 | 103 tok/s 73 tok/s 43 tok/s | ~547 ms ~551 ms ~560 ms | |
#231 | Qwen2-7B | 7B | INT4 Q8 FP16 | 143 tok/s 89 tok/s 49 tok/s | ~544 ms ~548 ms ~558 ms | |
#232 | Phi-3-small | 7B | INT4 Q8 FP16 | 143 tok/s 89 tok/s 49 tok/s | ~544 ms ~548 ms ~558 ms | |
#233 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 139 tok/s 86 tok/s 47 tok/s | ~566 ms ~571 ms ~580 ms | |
#234 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 421 tok/s 336 tok/s 232 tok/s | ~90 ms ~90 ms ~92 ms | |
#236 | Gemma 1 7B | 7B | INT4 Q8 FP16 | 103 tok/s 73 tok/s 43 tok/s | ~547 ms ~551 ms ~560 ms | |
#237 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 332 tok/s 236 tok/s 145 tok/s | ~171 ms ~172 ms ~175 ms | |
#238 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 114 tok/s 82 tok/s 49 tok/s | ~472 ms ~475 ms ~483 ms | |
#239 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 114 tok/s 82 tok/s 49 tok/s | ~472 ms ~475 ms ~483 ms | |
#240 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 114 tok/s 82 tok/s 49 tok/s | ~472 ms ~475 ms ~483 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 572 tok/s 519 tok/s 429 tok/s | ~28 ms ~28 ms ~28 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 522 tok/s 451 tok/s 346 tok/s | ~45 ms ~45 ms ~46 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 522 tok/s 451 tok/s 346 tok/s | ~45 ms ~45 ms ~46 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 513 tok/s 473 tok/s 397 tok/s | ~28 ms ~28 ms ~29 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 513 tok/s 473 tok/s 397 tok/s | ~28 ms ~28 ms ~29 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 499 tok/s 423 tok/s 315 tok/s | ~53 ms ~54 ms ~54 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 442 tok/s 348 tok/s 237 tok/s | ~90 ms ~90 ms ~92 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 421 tok/s 336 tok/s 232 tok/s | ~90 ms ~90 ms ~92 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 421 tok/s 336 tok/s 232 tok/s | ~90 ms ~90 ms ~92 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 359 tok/s 271 tok/s 176 tok/s | ~130 ms ~131 ms ~133 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 359 tok/s 271 tok/s 176 tok/s | ~130 ms ~131 ms ~133 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 339 tok/s 252 tok/s 160 tok/s | ~147 ms ~148 ms ~150 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 330 tok/s 243 tok/s 153 tok/s | ~154 ms ~155 ms ~158 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 267 tok/s 179 tok/s 104 tok/s | ~245 ms ~247 ms ~251 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 262 tok/s 222 tok/s 160 tok/s | ~115 ms ~115 ms ~117 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 262 tok/s 222 tok/s 160 tok/s | ~115 ms ~115 ms ~117 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 262 tok/s 222 tok/s 160 tok/s | ~115 ms ~115 ms ~117 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 Q8 FP16 | 257 tok/s 186 tok/s 103 tok/s | ~240 ms ~241 ms ~287 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 256 tok/s 175 tok/s 103 tok/s | ~245 ms ~247 ms ~251 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 253 tok/s 173 tok/s 102 tok/s | ~245 ms ~247 ms ~251 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 253 tok/s 173 tok/s 102 tok/s | ~245 ms ~247 ms ~251 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 253 tok/s 173 tok/s 102 tok/s | ~245 ms ~247 ms ~251 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 253 tok/s 173 tok/s 102 tok/s | ~245 ms ~247 ms ~251 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 253 tok/s 173 tok/s 102 tok/s | ~245 ms ~247 ms ~251 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 253 tok/s 173 tok/s 102 tok/s | ~245 ms ~247 ms ~251 ms | |
- | LFM2.5 24B A2B | 24B (2.5B active) | INT4 | 237 tok/s | ~271 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 Q8 | 222 tok/s 143 tok/s | ~299 ms ~354 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 Q8 | 222 tok/s 143 tok/s | ~299 ms ~354 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 | 215 tok/s | ~315 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 | 215 tok/s | ~315 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 212 tok/s 140 tok/s 80 tok/s | ~320 ms ~322 ms ~328 ms | |
- | Hy-Memory 30B A3B | 30B (3B active) | INT4 | 212 tok/s | ~320 ms | |
- | Hy-MT2 30B A3B | 30B (3B active) | INT4 | 212 tok/s | ~320 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 212 tok/s 140 tok/s 80 tok/s | ~320 ms ~322 ms ~328 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 204 tok/s 167 tok/s 116 tok/s | ~173 ms ~174 ms ~177 ms | |
- | MiniCPM-V 4.7 35B A3B | 35B (3B active) | INT4 | 199 tok/s | ~354 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 180 tok/s 142 tok/s 94 tok/s | ~225 ms ~226 ms ~230 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 171 tok/s 133 tok/s 87 tok/s | ~247 ms ~249 ms ~253 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 171 tok/s 133 tok/s 87 tok/s | ~247 ms ~249 ms ~253 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 149 tok/s 91 tok/s 49 tok/s | ~544 ms ~548 ms ~558 ms | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 143 tok/s 89 tok/s 49 tok/s | ~544 ms ~548 ms ~558 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 143 tok/s 89 tok/s 49 tok/s | ~544 ms ~548 ms ~558 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 139 tok/s 86 tok/s 47 tok/s | ~566 ms ~571 ms ~580 ms | |
- | Youtu Parsing Omni | 8B | INT4 Q8 FP16 | 132 tok/s 80 tok/s 43 tok/s | ~619 ms ~624 ms ~634 ms | |
- | LensVLM 9B | 9B | INT4 Q8 FP16 | 117 tok/s 72 tok/s 36 tok/s | ~695 ms ~700 ms ~773 ms | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 FP16 | 117 tok/s 72 tok/s 36 tok/s | ~695 ms ~700 ms ~773 ms | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 114 tok/s 82 tok/s 49 tok/s | ~472 ms ~475 ms ~483 ms | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 114 tok/s 82 tok/s 49 tok/s | ~472 ms ~475 ms ~483 ms | |
- | Falcon3-10B | 10B | INT4 Q8 FP16 | 108 tok/s 65 tok/s 33 tok/s | ~770 ms ~776 ms ~857 ms | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 Q8 | 108 tok/s 89 tok/s | ~292 ms ~319 ms | |
- | Falcon2-11B | 11B | INT4 Q8 FP16 | 104 tok/s 61 tok/s 29 tok/s | ~844 ms ~851 ms ~1.0s | |
- | Hunyuan Lite | 7B | INT4 Q8 FP16 | 103 tok/s 73 tok/s 43 tok/s | ~547 ms ~551 ms ~560 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 FP16 | 103 tok/s 73 tok/s 43 tok/s | ~547 ms ~551 ms ~560 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 FP16 | 102 tok/s 72 tok/s 43 tok/s | ~554 ms ~558 ms ~568 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 FP16 | 102 tok/s 72 tok/s 43 tok/s | ~554 ms ~558 ms ~568 ms | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 FP16 | 94 tok/s 65 tok/s 37 tok/s | ~622 ms ~626 ms ~691 ms | |
- | Typhoon-2-8B | 8B | INT4 Q8 FP16 | 94 tok/s 65 tok/s 37 tok/s | ~622 ms ~626 ms ~691 ms | |
- | GLM-4-9B | 9B | INT4 Q8 FP16 | 85 tok/s 59 tok/s 33 tok/s | ~698 ms ~703 ms ~776 ms | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 FP16 | 85 tok/s 59 tok/s 33 tok/s | ~698 ms ~703 ms ~776 ms | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 FP16 | 85 tok/s 59 tok/s 33 tok/s | ~698 ms ~703 ms ~776 ms | |
- | Yi-9B | 9B | INT4 Q8 FP16 | 85 tok/s 59 tok/s 33 tok/s | ~698 ms ~703 ms ~776 ms | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 FP16 | 84 tok/s 58 tok/s 32 tok/s | ~713 ms ~718 ms ~793 ms | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 FP16 | 84 tok/s 58 tok/s 32 tok/s | ~713 ms ~718 ms ~793 ms | |
- | GreenMind-14B-R1 | 14B | INT4 Q8 | 59 tok/s 40 tok/s | ~1.1s ~1.1s | |
- | AREX-2 | 27B | INT4 | 45 tok/s | ~2.0s | |
- | PPLX Decider v1 27B | 27B | INT4 | 45 tok/s | ~2.0s | |
- | PPLX Decider v1.1 27B | 27B | INT4 | 45 tok/s | ~2.0s | |
- | Agnes 3.0 Flash | 33B | INT4 | 38 tok/s | ~2.5s | |
- | Falcon-40B | 40B | INT4 | 31 tok/s | ~3.2s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
32B Models
8-bit
14B Models
16-bit
8B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
32B Models
LoRA
14B Models
Full Parameter Training
3B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA A30 (24GB).
Inference Sweet Spot
Optimized for 14B models at uncompressed or Q8 precision, and 32B models at Q4 with up to 32k context.
Bandwidth & Speed Profile
With 933 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 207 tok/s on an 8B Q4 model and 25 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning