AMD RX 9070 XT (16GB)
The AMD RX 9070 XT (16GB) provides 16 GB of memory with 640 GB/s bandwidth, supporting local inference via ROCm and Vulkan acceleration.
Memory
16 GB
Usable Memory Ceiling
16 GB
Bandwidth
640 GB/s
Max Inference Class
14B Models (Q4)
TDP
304W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on AMD RX 9070 XT (16GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on AMD RX 9070 XT (16GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#48 | Qwen 3.8 27B | 27B | INT4 | 28 tok/s | ~2.1s | |
#93 | Qwen3.5-27B | 27B | INT4 | 28 tok/s | ~2.1s | |
#111 | MiMo V2 Flash | 15B (309B active) | INT4 | 3 tok/s | ~20.8s | |
#112 | Phi-4 Reasoning Plus | 14B | INT4 | 41 tok/s | ~935 ms | |
#117 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 | 111 tok/s | ~349 ms | |
#125 | Qwen3.5-9B | 9B | INT4 Q8 | 76 tok/s 48 tok/s | ~605 ms ~612 ms | |
#128 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 91 tok/s 59 tok/s 30 tok/s | ~475 ms ~481 ms ~577 ms | |
#135 | Gemma 4 12B | 11.95B | INT4 Q8 | 46 tok/s 29 tok/s | ~802 ms ~951 ms | |
#140 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 126 tok/s 89 tok/s 54 tok/s | ~282 ms ~285 ms ~293 ms | |
#143 | GPT-OSS 20B | 21B (3.6B active) | INT4 | 56 tok/s | ~361 ms | |
#144 | GLM-4V | 9B | INT4 Q8 | 57 tok/s 38 tok/s | ~609 ms ~668 ms | |
#150 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 27 tok/s 16 tok/s 9 tok/s | ~1.9s ~1.9s ~2.0s | |
#158 | Granite 4.2 8B | 8B | INT4 Q8 | 83 tok/s 53 tok/s | ~539 ms ~546 ms | |
#159 | Gemma 4 E4B | 8B | INT4 Q8 | 83 tok/s 53 tok/s | ~539 ms ~546 ms | |
#166 | Phi-4 | 14B | INT4 Q8 | 54 tok/s 30 tok/s | ~929 ms ~1.1s | |
#167 | Gemma 3 27B | 27B | INT4 | 28 tok/s | ~2.1s | |
#178 | OLMo 3 7B Think | 7B | INT4 Q8 | 68 tok/s 49 tok/s | ~479 ms ~484 ms | |
#179 | DeepSeek-R1 14B | 14B | INT4 Q8 | 56 tok/s 30 tok/s | ~929 ms ~1.1s | |
#183 | Ministral 3 14B | 14B | INT4 | 41 tok/s | ~935 ms | |
#186 | Gemma 3 12B | 12B | INT4 Q8 | 61 tok/s 36 tok/s | ~800 ms ~878 ms | |
#188 | Mistral-Small-2501 | 24B | INT4 | 33 tok/s | ~1.7s | |
#189 | Ministral 3 8B | 8B | INT4 Q8 | 62 tok/s 45 tok/s | ~543 ms ~550 ms | |
#191 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 171 tok/s 134 tok/s 90 tok/s | ~154 ms ~155 ms ~159 ms | |
#192 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 171 tok/s 134 tok/s 90 tok/s | ~154 ms ~155 ms ~159 ms | |
#193 | Gemma 2 27B | 27B | INT4 | 28 tok/s | ~2.1s | |
#194 | Qwen2.5-14B | 14B | INT4 Q8 | 54 tok/s 30 tok/s | ~929 ms ~1.1s | |
#200 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 110 tok/s 75 tok/s 44 tok/s | ~353 ms ~357 ms ~366 ms | |
#202 | DeepSeek-R1 8B | 8B | INT4 Q8 | 85 tok/s 54 tok/s | ~539 ms ~546 ms | |
#204 | Qwen3-4B | 4B | INT4 Q8 FP16 | 126 tok/s 89 tok/s 54 tok/s | ~282 ms ~285 ms ~293 ms | |
#205 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 106 tok/s 85 tok/s 58 tok/s | ~220 ms ~222 ms ~228 ms | |
#207 | Gemma 2 9B | 9B | INT4 Q8 | 76 tok/s 48 tok/s | ~605 ms ~612 ms | |
#208 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 225 tok/s 196 tok/s 152 tok/s | ~68 ms ~68 ms ~70 ms | |
#209 | Qwen3-14B | 14B | INT4 Q8 | 54 tok/s 30 tok/s | ~929 ms ~1.1s | |
#210 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 126 tok/s 89 tok/s 54 tok/s | ~282 ms ~285 ms ~293 ms | |
#211 | Qwen3-8B | 8B | INT4 Q8 | 83 tok/s 53 tok/s | ~539 ms ~546 ms | |
#213 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 93 tok/s 60 tok/s 32 tok/s | ~475 ms ~481 ms ~535 ms | |
#214 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 182 tok/s 145 tok/s 100 tok/s | ~133 ms ~134 ms ~137 ms | |
#215 | Llama 3.1 8B | 8B | INT4 Q8 | 83 tok/s 53 tok/s | ~539 ms ~546 ms | |
#216 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 247 tok/s 235 tok/s 211 tok/s | ~28 ms ~29 ms ~29 ms | |
#218 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 238 tok/s 213 tok/s 173 tok/s | ~52 ms ~53 ms ~54 ms | |
#219 | Llama 3 8B | 8B | INT4 Q8 | 83 tok/s 53 tok/s | ~539 ms ~546 ms | |
#221 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 130 tok/s 92 tok/s 56 tok/s | ~269 ms ~272 ms ~279 ms | |
#223 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 91 tok/s 59 tok/s 30 tok/s | ~475 ms ~481 ms ~577 ms | |
#224 | Ministral-8B-2410 | 8B | INT4 Q8 | 83 tok/s 53 tok/s | ~539 ms ~546 ms | |
#225 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 190 tok/s 154 tok/s 108 tok/s | ~118 ms ~120 ms ~122 ms | |
#226 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 146 tok/s 107 tok/s 67 tok/s | ~218 ms ~220 ms ~226 ms | |
#228 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 130 tok/s 92 tok/s 56 tok/s | ~269 ms ~272 ms ~279 ms | |
#229 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 106 tok/s 92 tok/s 66 tok/s | ~167 ms ~168 ms ~186 ms | |
#230 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 88 tok/s 57 tok/s 29 tok/s | ~494 ms ~500 ms ~601 ms | |
#231 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 213 tok/s 181 tok/s 136 tok/s | ~84 ms ~84 ms ~86 ms | |
#232 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 171 tok/s 134 tok/s 90 tok/s | ~154 ms ~155 ms ~159 ms | |
#233 | Phi-3-medium | 14B | INT4 Q8 | 54 tok/s 30 tok/s | ~929 ms ~1.1s | |
#234 | OLMo 3 7B Instruct | 7B | INT4 Q8 | 68 tok/s 49 tok/s | ~479 ms ~484 ms | |
#236 | Qwen2-7B | 7B | INT4 Q8 FP16 | 91 tok/s 59 tok/s 30 tok/s | ~475 ms ~481 ms ~577 ms | |
#237 | Phi-3-small | 7B | INT4 Q8 FP16 | 91 tok/s 59 tok/s 30 tok/s | ~475 ms ~481 ms ~577 ms | |
#238 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 213 tok/s 181 tok/s 136 tok/s | ~84 ms ~84 ms ~86 ms | |
#239 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 88 tok/s 57 tok/s 29 tok/s | ~494 ms ~500 ms ~601 ms | |
#241 | Gemma 1 7B | 7B | INT4 Q8 | 68 tok/s 49 tok/s | ~479 ms ~484 ms | |
#242 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 179 tok/s 138 tok/s 92 tok/s | ~153 ms ~155 ms ~159 ms | |
#243 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 74 tok/s 55 tok/s 33 tok/s | ~414 ms ~419 ms ~466 ms | |
#244 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 74 tok/s 55 tok/s 33 tok/s | ~414 ms ~419 ms ~466 ms | |
#245 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 74 tok/s 55 tok/s 33 tok/s | ~414 ms ~419 ms ~466 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 260 tok/s 245 tok/s 216 tok/s | ~30 ms ~31 ms ~31 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 246 tok/s 223 tok/s 185 tok/s | ~45 ms ~45 ms ~46 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 246 tok/s 223 tok/s 185 tok/s | ~45 ms ~45 ms ~46 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 243 tok/s 230 tok/s 204 tok/s | ~31 ms ~31 ms ~32 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 243 tok/s 230 tok/s 204 tok/s | ~31 ms ~31 ms ~32 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 238 tok/s 213 tok/s 173 tok/s | ~52 ms ~53 ms ~54 ms | |
- | PixArt-α XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 238 tok/s 212 tok/s 172 tok/s | ~53 ms ~54 ms ~55 ms | |
- | PixArt-Σ XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 238 tok/s 212 tok/s 172 tok/s | ~53 ms ~54 ms ~55 ms | |
- | Nemotron 3.5 Parse | 0.63B | INT4 Q8 FP16 | 236 tok/s 210 tok/s 169 tok/s | ~55 ms ~55 ms ~56 ms | |
- | SD 2.1 Turbo | 0.86B | INT4 Q8 FP16 | 221 tok/s 191 tok/s 147 tok/s | ~72 ms ~73 ms ~75 ms | |
- | Stable Diffusion 1.4 | 0.86B | INT4 Q8 FP16 | 221 tok/s 191 tok/s 147 tok/s | ~72 ms ~73 ms ~75 ms | |
- | Stable Diffusion 2.1 | 0.86B | INT4 Q8 FP16 | 221 tok/s 191 tok/s 147 tok/s | ~72 ms ~73 ms ~75 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 220 tok/s 186 tok/s 138 tok/s | ~83 ms ~84 ms ~86 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 213 tok/s 181 tok/s 136 tok/s | ~84 ms ~84 ms ~86 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 213 tok/s 181 tok/s 136 tok/s | ~84 ms ~84 ms ~86 ms | |
- | Kandinsky 2.2 Decoder | 1.22B | INT4 Q8 FP16 | 202 tok/s 168 tok/s 122 tok/s | ~99 ms ~100 ms ~102 ms | |
- | Wan 2.1 T2V 1.3B | 1.3B | INT4 Q8 FP16 | 199 tok/s 164 tok/s 118 tok/s | ~104 ms ~105 ms ~108 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 190 tok/s 154 tok/s 108 tok/s | ~118 ms ~120 ms ~122 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 190 tok/s 154 tok/s 108 tok/s | ~118 ms ~120 ms ~122 ms | |
- | SVD Image-to-Video XT | 1.5B | INT4 Q8 FP16 | 190 tok/s 154 tok/s 108 tok/s | ~118 ms ~120 ms ~122 ms | |
- | SVD Image-to-Video XT 1.1 | 1.5B | INT4 Q8 FP16 | 190 tok/s 154 tok/s 108 tok/s | ~118 ms ~120 ms ~122 ms | |
- | LFM3 1.7B | 1.7B | INT4 Q8 FP16 | 182 tok/s 145 tok/s 100 tok/s | ~133 ms ~134 ms ~137 ms | |
- | LFM3 1.7B Base | 1.7B | INT4 Q8 FP16 | 182 tok/s 145 tok/s 100 tok/s | ~133 ms ~134 ms ~137 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 182 tok/s 145 tok/s 100 tok/s | ~133 ms ~134 ms ~137 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 179 tok/s 141 tok/s 96 tok/s | ~139 ms ~141 ms ~144 ms | |
- | Moondream 3.1 | 9B (2B active) | INT4 Q8 | 172 tok/s 134 tok/s | ~172 ms ~174 ms | |
- | CogVideoX 2B | 2B | INT4 Q8 FP16 | 171 tok/s 134 tok/s 90 tok/s | ~154 ms ~155 ms ~159 ms | |
- | LFM3 Audio 2B | 2B | INT4 Q8 FP16 | 171 tok/s 134 tok/s 90 tok/s | ~154 ms ~155 ms ~159 ms | |
- | LFM3 Omni 2B | 2B | INT4 Q8 FP16 | 171 tok/s 134 tok/s 90 tok/s | ~154 ms ~155 ms ~159 ms | |
- | LFM3 VL 2B | 2B | INT4 Q8 FP16 | 171 tok/s 134 tok/s 90 tok/s | ~154 ms ~155 ms ~159 ms | |
- | Liquid D1 2B | 2B | INT4 Q8 FP16 | 171 tok/s 134 tok/s 90 tok/s | ~154 ms ~155 ms ~159 ms | |
- | LTX-Video 0.9.1 | 2B | INT4 Q8 FP16 | 171 tok/s 134 tok/s 90 tok/s | ~154 ms ~155 ms ~159 ms | |
- | Stable Diffusion 3 Medium | 2B | INT4 Q8 FP16 | 171 tok/s 134 tok/s 90 tok/s | ~154 ms ~155 ms ~159 ms | |
- | Kandinsky 3.0 | 2.1B | INT4 Q8 FP16 | 168 tok/s 130 tok/s 87 tok/s | ~160 ms ~162 ms ~166 ms | |
- | LTX-2 | 2.2B | INT4 Q8 FP16 | 166 tok/s 127 tok/s 84 tok/s | ~166 ms ~168 ms ~172 ms | |
- | SDXL 1.0 Refiner | 2.31B | INT4 Q8 FP16 | 163 tok/s 124 tok/s 81 tok/s | ~174 ms ~175 ms ~180 ms | |
- | Stable Diffusion 3.5 Medium | 2.5B | INT4 Q8 FP16 | 157 tok/s 119 tok/s 77 tok/s | ~186 ms ~188 ms ~192 ms | |
- | SDXL 1.0 Base | 2.56B | INT4 Q8 FP16 | 156 tok/s 117 tok/s 76 tok/s | ~189 ms ~192 ms ~196 ms | |
- | SDXL Turbo | 2.56B | INT4 Q8 FP16 | 156 tok/s 117 tok/s 76 tok/s | ~189 ms ~192 ms ~196 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 152 tok/s 110 tok/s 68 tok/s | ~217 ms ~220 ms ~225 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 150 tok/s 131 tok/s 100 tok/s | ~106 ms ~107 ms ~109 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 150 tok/s 131 tok/s 100 tok/s | ~106 ms ~107 ms ~109 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 150 tok/s 131 tok/s 100 tok/s | ~106 ms ~107 ms ~109 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 147 tok/s 108 tok/s 68 tok/s | ~218 ms ~220 ms ~226 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 Q8 | 147 tok/s 108 tok/s | ~213 ms ~232 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 146 tok/s 107 tok/s 67 tok/s | ~218 ms ~220 ms ~226 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 146 tok/s 107 tok/s 67 tok/s | ~218 ms ~220 ms ~226 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 146 tok/s 107 tok/s 67 tok/s | ~218 ms ~220 ms ~226 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 146 tok/s 107 tok/s 67 tok/s | ~218 ms ~220 ms ~226 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 146 tok/s 107 tok/s 67 tok/s | ~218 ms ~220 ms ~226 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 146 tok/s 107 tok/s 67 tok/s | ~218 ms ~220 ms ~226 ms | |
- | LFM2.5 24B A2B | 24B (2.5B active) | INT4 | 131 tok/s | ~260 ms | |
- | FLUX.2 Klein 4B | 4B | INT4 Q8 FP16 | 126 tok/s 89 tok/s 54 tok/s | ~282 ms ~285 ms ~293 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 126 tok/s 89 tok/s 54 tok/s | ~282 ms ~285 ms ~293 ms | |
- | Mage-VL Cowork | 4B | INT4 Q8 FP16 | 126 tok/s 89 tok/s 54 tok/s | ~282 ms ~285 ms ~293 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 126 tok/s 89 tok/s 54 tok/s | ~282 ms ~285 ms ~293 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 | 125 tok/s | ~286 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 | 125 tok/s | ~286 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 122 tok/s 104 tok/s 75 tok/s | ~156 ms ~157 ms ~161 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 | 115 tok/s | ~324 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 | 115 tok/s | ~324 ms | |
- | CogVideoX 1.5 5B | 5B | INT4 Q8 FP16 | 112 tok/s 76 tok/s 45 tok/s | ~346 ms ~351 ms ~360 ms | |
- | CogVideoX 5B | 5B | INT4 Q8 FP16 | 112 tok/s 76 tok/s 45 tok/s | ~346 ms ~351 ms ~360 ms | |
- | Wan 2.2 TI2V 5B | 5B | INT4 Q8 FP16 | 112 tok/s 76 tok/s 45 tok/s | ~346 ms ~351 ms ~360 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 110 tok/s 90 tok/s 62 tok/s | ~201 ms ~203 ms ~208 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 106 tok/s 85 tok/s 58 tok/s | ~220 ms ~222 ms ~228 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 106 tok/s 85 tok/s 58 tok/s | ~220 ms ~222 ms ~228 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 94 tok/s 61 tok/s 32 tok/s | ~475 ms ~481 ms ~535 ms | |
- | AuraFlow v0.2 | 6.8B | INT4 Q8 FP16 | 92 tok/s 61 tok/s 33 tok/s | ~462 ms ~468 ms ~520 ms | |
- | AuraFlow v0.3 | 6.8B | INT4 Q8 FP16 | 92 tok/s 61 tok/s 33 tok/s | ~462 ms ~468 ms ~520 ms | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 91 tok/s 59 tok/s 30 tok/s | ~475 ms ~481 ms ~577 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 91 tok/s 59 tok/s 30 tok/s | ~475 ms ~481 ms ~577 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 88 tok/s 57 tok/s 29 tok/s | ~494 ms ~500 ms ~601 ms | |
- | Youtu Parsing Omni | 8B | INT4 Q8 | 85 tok/s 54 tok/s | ~539 ms ~546 ms | |
- | Stable Diffusion 3.5 Large | 8.1B | INT4 Q8 | 82 tok/s 53 tok/s | ~546 ms ~553 ms | |
- | Stable Diffusion 3.5 Large Turbo | 8.1B | INT4 Q8 | 82 tok/s 53 tok/s | ~546 ms ~553 ms | |
- | FLUX.2 Klein 9B | 9B | INT4 Q8 | 76 tok/s 48 tok/s | ~605 ms ~612 ms | |
- | LensVLM 9B | 9B | INT4 Q8 | 76 tok/s 48 tok/s | ~605 ms ~612 ms | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 | 76 tok/s 48 tok/s | ~605 ms ~612 ms | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 74 tok/s 55 tok/s 33 tok/s | ~414 ms ~419 ms ~466 ms | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 74 tok/s 55 tok/s 33 tok/s | ~414 ms ~419 ms ~466 ms | |
- | Falcon3-10B | 10B | INT4 Q8 | 71 tok/s 44 tok/s | ~669 ms ~678 ms | |
- | Mochi 1 Preview | 10B | INT4 Q8 | 71 tok/s 44 tok/s | ~669 ms ~678 ms | |
- | Falcon2-11B | 11B | INT4 Q8 | 68 tok/s 42 tok/s | ~733 ms ~743 ms | |
- | Hunyuan Lite | 7B | INT4 Q8 | 68 tok/s 49 tok/s | ~479 ms ~484 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 | 68 tok/s 49 tok/s | ~479 ms ~484 ms | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 | 67 tok/s | ~282 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 | 67 tok/s 49 tok/s | ~485 ms ~491 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 | 67 tok/s 49 tok/s | ~485 ms ~491 ms | |
- | FLUX.1 Canny Dev | 11.9B | INT4 Q8 | 62 tok/s 36 tok/s | ~794 ms ~871 ms | |
- | FLUX.1 Depth Dev | 11.9B | INT4 Q8 | 62 tok/s 36 tok/s | ~794 ms ~871 ms | |
- | FLUX.1 Dev | 11.9B | INT4 Q8 | 62 tok/s 36 tok/s | ~794 ms ~871 ms | |
- | FLUX.1 Fill Dev | 11.9B | INT4 Q8 | 62 tok/s 36 tok/s | ~794 ms ~871 ms | |
- | FLUX.1 Schnell | 11.9B | INT4 Q8 | 62 tok/s 36 tok/s | ~794 ms ~871 ms | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 | 62 tok/s 45 tok/s | ~543 ms ~550 ms | |
- | Typhoon-2-8B | 8B | INT4 Q8 | 62 tok/s 45 tok/s | ~543 ms ~550 ms | |
- | HunyuanVideo 1.5 720P T2V | 13B | INT4 Q8 | 58 tok/s 34 tok/s | ~865 ms ~950 ms | |
- | GLM-4-9B | 9B | INT4 Q8 | 57 tok/s 38 tok/s | ~609 ms ~668 ms | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 | 57 tok/s 38 tok/s | ~609 ms ~668 ms | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 | 57 tok/s 38 tok/s | ~609 ms ~668 ms | |
- | Yi-9B | 9B | INT4 Q8 | 57 tok/s 38 tok/s | ~609 ms ~668 ms | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 | 56 tok/s 38 tok/s | ~622 ms ~682 ms | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 | 56 tok/s 38 tok/s | ~622 ms ~682 ms | |
- | Wan 2.1 I2V 14B 720P | 14B | INT4 Q8 | 54 tok/s 30 tok/s | ~929 ms ~1.1s | |
- | Wan 2.1 T2V 14B | 14B | INT4 Q8 | 54 tok/s 30 tok/s | ~929 ms ~1.1s | |
- | Wan 2.2 T2V A14B | 14B | INT4 Q8 | 54 tok/s 30 tok/s | ~929 ms ~1.1s | |
- | FLUX.2 Dev | 15B | INT4 Q8 | 52 tok/s 28 tok/s | ~993 ms ~1.2s | |
- | GreenMind-14B-R1 | 14B | INT4 | 41 tok/s | ~935 ms | |
- | Arete 27B | 27B | INT4 | 28 tok/s | ~2.1s | |
- | AREX-2 | 27B | INT4 | 28 tok/s | ~2.1s | |
- | PPLX Decider v1 27B | 27B | INT4 | 28 tok/s | ~2.1s | |
- | PPLX Decider v1.1 27B | 27B | INT4 | 28 tok/s | ~2.1s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
14B Models
8-bit
8B Models
16-bit
3B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
14B Models
LoRA
8B Models
Full Parameter Training
1B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on AMD RX 9070 XT (16GB).
Inference Sweet Spot
Optimal for 8B models across extended context (32k+). 14B models fit comfortably at Q4 precision with standard context.
Bandwidth & Speed Profile
With 640 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 142 tok/s on an 8B Q4 model and 17 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning