NVIDIA RTX 4070 Ti SUPER (16GB)
The NVIDIA RTX 4070 Ti SUPER (16GB) features 16 GB of dedicated VRAM and 672 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
16 GB
Usable Memory Ceiling
16 GB
Bandwidth
672.3 GB/s
Max Inference Class
14B Models (Q4)
TDP
285W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA RTX 4070 Ti SUPER (16GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA RTX 4070 Ti SUPER (16GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#48 | Qwen 3.8 27B | 27B | INT4 | 30 tok/s | ~2.0s | |
#93 | Qwen3.5-27B | 27B | INT4 | 30 tok/s | ~2.0s | |
#111 | MiMo V2 Flash | 15B (309B active) | INT4 | 3 tok/s | ~20.0s | |
#112 | Phi-4 Reasoning Plus | 14B | INT4 | 44 tok/s | ~899 ms | |
#117 | Gemma 4 26B A4B | 25.2B (3.8B active) | INT4 | 126 tok/s | ~334 ms | |
#125 | Qwen3.5-9B | 9B | INT4 Q8 | 84 tok/s 53 tok/s | ~581 ms ~588 ms | |
#128 | K2 Horizon 7B | 7B | INT4 Q8 FP16 | 101 tok/s 65 tok/s 33 tok/s | ~456 ms ~461 ms ~554 ms | |
#135 | Gemma 4 12B | 11.95B | INT4 Q8 | 50 tok/s 31 tok/s | ~771 ms ~913 ms | |
#140 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 145 tok/s 99 tok/s 59 tok/s | ~270 ms ~273 ms ~280 ms | |
#143 | GPT-OSS 20B | 21B (3.6B active) | INT4 | 61 tok/s | ~346 ms | |
#144 | GLM-4V | 9B | INT4 Q8 | 62 tok/s 41 tok/s | ~585 ms ~642 ms | |
#150 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 29 tok/s 17 tok/s 9 tok/s | ~1.8s ~1.8s ~1.9s | |
#158 | Granite 4.2 8B | 8B | INT4 Q8 | 92 tok/s 58 tok/s | ~518 ms ~524 ms | |
#159 | Gemma 4 E4B | 8B | INT4 Q8 | 92 tok/s 58 tok/s | ~518 ms ~524 ms | |
#166 | Phi-4 | 14B | INT4 Q8 | 59 tok/s 32 tok/s | ~893 ms ~1.1s | |
#167 | Gemma 3 27B | 27B | INT4 | 30 tok/s | ~2.0s | |
#178 | OLMo 3 7B Think | 7B | INT4 Q8 | 74 tok/s 53 tok/s | ~459 ms ~465 ms | |
#179 | DeepSeek-R1 14B | 14B | INT4 Q8 | 61 tok/s 33 tok/s | ~892 ms ~1.1s | |
#183 | Ministral 3 14B | 14B | INT4 | 44 tok/s | ~899 ms | |
#186 | Gemma 3 12B | 12B | INT4 Q8 | 67 tok/s 39 tok/s | ~768 ms ~844 ms | |
#188 | Mistral-Small-2501 | 24B | INT4 | 36 tok/s | ~1.6s | |
#189 | Ministral 3 8B | 8B | INT4 Q8 | 68 tok/s 48 tok/s | ~522 ms ~528 ms | |
#191 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~146 ms ~148 ms ~152 ms | |
#192 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~146 ms ~148 ms ~152 ms | |
#193 | Gemma 2 27B | 27B | INT4 | 30 tok/s | ~2.0s | |
#194 | Qwen2.5-14B | 14B | INT4 Q8 | 59 tok/s 32 tok/s | ~893 ms ~1.1s | |
#200 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 125 tok/s 83 tok/s 48 tok/s | ~338 ms ~342 ms ~351 ms | |
#202 | DeepSeek-R1 8B | 8B | INT4 Q8 | 94 tok/s 59 tok/s | ~517 ms ~524 ms | |
#204 | Qwen3-4B | 4B | INT4 Q8 FP16 | 145 tok/s 99 tok/s 59 tok/s | ~270 ms ~273 ms ~280 ms | |
#205 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 120 tok/s 95 tok/s 63 tok/s | ~210 ms ~213 ms ~218 ms | |
#207 | Gemma 2 9B | 9B | INT4 Q8 | 84 tok/s 53 tok/s | ~581 ms ~588 ms | |
#208 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 278 tok/s 237 tok/s 178 tok/s | ~64 ms ~64 ms ~66 ms | |
#209 | Qwen3-14B | 14B | INT4 Q8 | 59 tok/s 32 tok/s | ~893 ms ~1.1s | |
#210 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 145 tok/s 99 tok/s 59 tok/s | ~270 ms ~273 ms ~280 ms | |
#211 | Qwen3-8B | 8B | INT4 Q8 | 92 tok/s 58 tok/s | ~518 ms ~524 ms | |
#213 | DeepSeek-R1 7B | 7B | INT4 Q8 FP16 | 104 tok/s 66 tok/s 35 tok/s | ~456 ms ~461 ms ~513 ms | |
#214 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 217 tok/s 169 tok/s 112 tok/s | ~126 ms ~128 ms ~131 ms | |
#215 | Llama 3.1 8B | 8B | INT4 Q8 | 92 tok/s 58 tok/s | ~518 ms ~524 ms | |
#216 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 311 tok/s 293 tok/s 257 tok/s | ~26 ms ~26 ms ~27 ms | |
#218 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 298 tok/s 261 tok/s 205 tok/s | ~49 ms ~49 ms ~50 ms | |
#219 | Llama 3 8B | 8B | INT4 Q8 | 92 tok/s 58 tok/s | ~518 ms ~524 ms | |
#221 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 149 tok/s 103 tok/s 61 tok/s | ~258 ms ~261 ms ~267 ms | |
#223 | Qwen2.5-7B | 7B | INT4 Q8 FP16 | 101 tok/s 65 tok/s 33 tok/s | ~456 ms ~461 ms ~554 ms | |
#224 | Ministral-8B-2410 | 8B | INT4 Q8 | 92 tok/s 58 tok/s | ~518 ms ~524 ms | |
#225 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 228 tok/s 180 tok/s 122 tok/s | ~113 ms ~114 ms ~116 ms | |
#226 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 169 tok/s 121 tok/s 74 tok/s | ~208 ms ~210 ms ~216 ms | |
#228 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 149 tok/s 103 tok/s 61 tok/s | ~258 ms ~261 ms ~267 ms | |
#229 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 FP16 | 120 tok/s 103 tok/s 72 tok/s | ~159 ms ~161 ms ~177 ms | |
#230 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 FP16 | 98 tok/s 63 tok/s 31 tok/s | ~474 ms ~480 ms ~576 ms | |
#231 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 260 tok/s 216 tok/s 157 tok/s | ~79 ms ~80 ms ~82 ms | |
#232 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~146 ms ~148 ms ~152 ms | |
#233 | Phi-3-medium | 14B | INT4 Q8 | 59 tok/s 32 tok/s | ~893 ms ~1.1s | |
#234 | OLMo 3 7B Instruct | 7B | INT4 Q8 | 74 tok/s 53 tok/s | ~459 ms ~465 ms | |
#236 | Qwen2-7B | 7B | INT4 Q8 FP16 | 101 tok/s 65 tok/s 33 tok/s | ~456 ms ~461 ms ~554 ms | |
#237 | Phi-3-small | 7B | INT4 Q8 FP16 | 101 tok/s 65 tok/s 33 tok/s | ~456 ms ~461 ms ~554 ms | |
#238 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 260 tok/s 216 tok/s 157 tok/s | ~79 ms ~80 ms ~82 ms | |
#239 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 FP16 | 98 tok/s 63 tok/s 31 tok/s | ~474 ms ~480 ms ~576 ms | |
#241 | Gemma 1 7B | 7B | INT4 Q8 | 74 tok/s 53 tok/s | ~459 ms ~465 ms | |
#242 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 214 tok/s 159 tok/s 102 tok/s | ~146 ms ~148 ms ~151 ms | |
#243 | ChatGLM3-6B | 6B | INT4 Q8 FP16 | 82 tok/s 60 tok/s 35 tok/s | ~397 ms ~402 ms ~446 ms | |
#244 | ChatGLM2-6B | 6B | INT4 Q8 FP16 | 82 tok/s 60 tok/s 35 tok/s | ~397 ms ~402 ms ~446 ms | |
#245 | ChatGLM-6B | 6B | INT4 Q8 FP16 | 82 tok/s 60 tok/s 35 tok/s | ~397 ms ~402 ms ~446 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 331 tok/s 307 tok/s 264 tok/s | ~28 ms ~28 ms ~29 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 308 tok/s 275 tok/s 222 tok/s | ~42 ms ~42 ms ~43 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 308 tok/s 275 tok/s 222 tok/s | ~42 ms ~42 ms ~43 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 304 tok/s 285 tok/s 248 tok/s | ~28 ms ~28 ms ~29 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 304 tok/s 285 tok/s 248 tok/s | ~28 ms ~28 ms ~29 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 298 tok/s 261 tok/s 205 tok/s | ~49 ms ~49 ms ~50 ms | |
- | PixArt-α XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 297 tok/s 260 tok/s 203 tok/s | ~50 ms ~50 ms ~51 ms | |
- | PixArt-Σ XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 297 tok/s 260 tok/s 203 tok/s | ~50 ms ~50 ms ~51 ms | |
- | Nemotron 3.5 Parse | 0.63B | INT4 Q8 FP16 | 294 tok/s 257 tok/s 200 tok/s | ~51 ms ~52 ms ~53 ms | |
- | SD 2.1 Turbo | 0.86B | INT4 Q8 FP16 | 273 tok/s 230 tok/s 171 tok/s | ~68 ms ~69 ms ~70 ms | |
- | Stable Diffusion 1.4 | 0.86B | INT4 Q8 FP16 | 273 tok/s 230 tok/s 171 tok/s | ~68 ms ~69 ms ~70 ms | |
- | Stable Diffusion 2.1 | 0.86B | INT4 Q8 FP16 | 273 tok/s 230 tok/s 171 tok/s | ~68 ms ~69 ms ~70 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 270 tok/s 223 tok/s 160 tok/s | ~79 ms ~80 ms ~81 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 260 tok/s 216 tok/s 157 tok/s | ~79 ms ~80 ms ~82 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 260 tok/s 216 tok/s 157 tok/s | ~79 ms ~80 ms ~82 ms | |
- | Kandinsky 2.2 Decoder | 1.22B | INT4 Q8 FP16 | 245 tok/s 199 tok/s 139 tok/s | ~94 ms ~95 ms ~97 ms | |
- | Wan 2.1 T2V 1.3B | 1.3B | INT4 Q8 FP16 | 240 tok/s 193 tok/s 134 tok/s | ~99 ms ~100 ms ~102 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 228 tok/s 180 tok/s 122 tok/s | ~113 ms ~114 ms ~116 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 228 tok/s 180 tok/s 122 tok/s | ~113 ms ~114 ms ~116 ms | |
- | SVD Image-to-Video XT | 1.5B | INT4 Q8 FP16 | 228 tok/s 180 tok/s 122 tok/s | ~113 ms ~114 ms ~116 ms | |
- | SVD Image-to-Video XT 1.1 | 1.5B | INT4 Q8 FP16 | 228 tok/s 180 tok/s 122 tok/s | ~113 ms ~114 ms ~116 ms | |
- | LFM3 1.7B | 1.7B | INT4 Q8 FP16 | 217 tok/s 169 tok/s 112 tok/s | ~126 ms ~128 ms ~131 ms | |
- | LFM3 1.7B Base | 1.7B | INT4 Q8 FP16 | 217 tok/s 169 tok/s 112 tok/s | ~126 ms ~128 ms ~131 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 217 tok/s 169 tok/s 112 tok/s | ~126 ms ~128 ms ~131 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 213 tok/s 164 tok/s 108 tok/s | ~133 ms ~134 ms ~137 ms | |
- | CogVideoX 2B | 2B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~146 ms ~148 ms ~152 ms | |
- | LFM3 Audio 2B | 2B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~146 ms ~148 ms ~152 ms | |
- | LFM3 Omni 2B | 2B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~146 ms ~148 ms ~152 ms | |
- | LFM3 VL 2B | 2B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~146 ms ~148 ms ~152 ms | |
- | Liquid D1 2B | 2B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~146 ms ~148 ms ~152 ms | |
- | LTX-Video 0.9.1 | 2B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~146 ms ~148 ms ~152 ms | |
- | Moondream 3.1 | 9B (2B active) | INT4 Q8 | 203 tok/s 154 tok/s | ~164 ms ~166 ms | |
- | Stable Diffusion 3 Medium | 2B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~146 ms ~148 ms ~152 ms | |
- | Kandinsky 3.0 | 2.1B | INT4 Q8 FP16 | 199 tok/s 150 tok/s 97 tok/s | ~153 ms ~154 ms ~158 ms | |
- | LTX-2 | 2.2B | INT4 Q8 FP16 | 195 tok/s 146 tok/s 93 tok/s | ~159 ms ~160 ms ~164 ms | |
- | SDXL 1.0 Refiner | 2.31B | INT4 Q8 FP16 | 191 tok/s 142 tok/s 90 tok/s | ~166 ms ~167 ms ~171 ms | |
- | Stable Diffusion 3.5 Medium | 2.5B | INT4 Q8 FP16 | 185 tok/s 135 tok/s 85 tok/s | ~177 ms ~179 ms ~184 ms | |
- | SDXL 1.0 Base | 2.56B | INT4 Q8 FP16 | 183 tok/s 134 tok/s 84 tok/s | ~181 ms ~183 ms ~187 ms | |
- | SDXL Turbo | 2.56B | INT4 Q8 FP16 | 183 tok/s 134 tok/s 84 tok/s | ~181 ms ~183 ms ~187 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 177 tok/s 125 tok/s 75 tok/s | ~208 ms ~210 ms ~215 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 175 tok/s 151 tok/s 112 tok/s | ~100 ms ~101 ms ~104 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 175 tok/s 151 tok/s 112 tok/s | ~100 ms ~101 ms ~104 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 175 tok/s 151 tok/s 112 tok/s | ~100 ms ~101 ms ~104 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 Q8 | 172 tok/s 122 tok/s | ~203 ms ~222 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 171 tok/s 122 tok/s 74 tok/s | ~208 ms ~210 ms ~216 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 169 tok/s 121 tok/s 74 tok/s | ~208 ms ~210 ms ~216 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 169 tok/s 121 tok/s 74 tok/s | ~208 ms ~210 ms ~216 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 169 tok/s 121 tok/s 74 tok/s | ~208 ms ~210 ms ~216 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 169 tok/s 121 tok/s 74 tok/s | ~208 ms ~210 ms ~216 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 169 tok/s 121 tok/s 74 tok/s | ~208 ms ~210 ms ~216 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 169 tok/s 121 tok/s 74 tok/s | ~208 ms ~210 ms ~216 ms | |
- | LFM2.5 24B A2B | 24B (2.5B active) | INT4 | 152 tok/s | ~249 ms | |
- | FLUX.2 Klein 4B | 4B | INT4 Q8 FP16 | 145 tok/s 99 tok/s 59 tok/s | ~270 ms ~273 ms ~280 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 145 tok/s 99 tok/s 59 tok/s | ~270 ms ~273 ms ~280 ms | |
- | Mage-VL Cowork | 4B | INT4 Q8 FP16 | 145 tok/s 99 tok/s 59 tok/s | ~270 ms ~273 ms ~280 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 145 tok/s 99 tok/s 59 tok/s | ~270 ms ~273 ms ~280 ms | |
- | ERNIE-4.5-21B-A3B | 21B (3B active) | INT4 | 143 tok/s | ~274 ms | |
- | ERNIE-4.5-21B-A3B-Base | 21B (3B active) | INT4 | 143 tok/s | ~274 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 140 tok/s 117 tok/s 83 tok/s | ~149 ms ~150 ms ~154 ms | |
- | ERNIE-4.5-VL-28B-A3B | 28B (3B active) | INT4 | 132 tok/s | ~311 ms | |
- | ERNIE-4.5-VL-28B-A3B-Base | 28B (3B active) | INT4 | 132 tok/s | ~311 ms | |
- | CogVideoX 1.5 5B | 5B | INT4 Q8 FP16 | 127 tok/s 84 tok/s 49 tok/s | ~332 ms ~336 ms ~345 ms | |
- | CogVideoX 5B | 5B | INT4 Q8 FP16 | 127 tok/s 84 tok/s 49 tok/s | ~332 ms ~336 ms ~345 ms | |
- | Wan 2.2 TI2V 5B | 5B | INT4 Q8 FP16 | 127 tok/s 84 tok/s 49 tok/s | ~332 ms ~336 ms ~345 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 125 tok/s 101 tok/s 68 tok/s | ~192 ms ~194 ms ~199 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 120 tok/s 95 tok/s 63 tok/s | ~210 ms ~213 ms ~218 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 120 tok/s 95 tok/s 63 tok/s | ~210 ms ~213 ms ~218 ms | |
- | Falcon-7B | 7B | INT4 Q8 FP16 | 105 tok/s 66 tok/s 35 tok/s | ~456 ms ~461 ms ~513 ms | |
- | AuraFlow v0.2 | 6.8B | INT4 Q8 FP16 | 103 tok/s 66 tok/s 35 tok/s | ~444 ms ~449 ms ~499 ms | |
- | AuraFlow v0.3 | 6.8B | INT4 Q8 FP16 | 103 tok/s 66 tok/s 35 tok/s | ~444 ms ~449 ms ~499 ms | |
- | Falcon3-7B | 7B | INT4 Q8 FP16 | 101 tok/s 65 tok/s 33 tok/s | ~456 ms ~461 ms ~554 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 FP16 | 101 tok/s 65 tok/s 33 tok/s | ~456 ms ~461 ms ~554 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 FP16 | 98 tok/s 63 tok/s 31 tok/s | ~474 ms ~480 ms ~576 ms | |
- | Youtu Parsing Omni | 8B | INT4 Q8 | 94 tok/s 59 tok/s | ~517 ms ~524 ms | |
- | Stable Diffusion 3.5 Large | 8.1B | INT4 Q8 | 91 tok/s 58 tok/s | ~524 ms ~530 ms | |
- | Stable Diffusion 3.5 Large Turbo | 8.1B | INT4 Q8 | 91 tok/s 58 tok/s | ~524 ms ~530 ms | |
- | FLUX.2 Klein 9B | 9B | INT4 Q8 | 84 tok/s 53 tok/s | ~581 ms ~588 ms | |
- | LensVLM 9B | 9B | INT4 Q8 | 84 tok/s 53 tok/s | ~581 ms ~588 ms | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 | 84 tok/s 53 tok/s | ~581 ms ~588 ms | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 FP16 | 82 tok/s 60 tok/s 35 tok/s | ~397 ms ~402 ms ~446 ms | |
- | Yi-6B | 6B | INT4 Q8 FP16 | 82 tok/s 60 tok/s 35 tok/s | ~397 ms ~402 ms ~446 ms | |
- | Falcon3-10B | 10B | INT4 Q8 | 78 tok/s 48 tok/s | ~643 ms ~651 ms | |
- | Mochi 1 Preview | 10B | INT4 Q8 | 78 tok/s 48 tok/s | ~643 ms ~651 ms | |
- | Falcon2-11B | 11B | INT4 Q8 | 75 tok/s 45 tok/s | ~704 ms ~713 ms | |
- | Hunyuan Lite | 7B | INT4 Q8 | 74 tok/s 53 tok/s | ~459 ms ~465 ms | |
- | Kimi-VL-A3B-Instruct | 16B (3B active) | INT4 | 74 tok/s | ~270 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 | 74 tok/s 53 tok/s | ~459 ms ~465 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 | 74 tok/s 53 tok/s | ~466 ms ~471 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 | 74 tok/s 53 tok/s | ~466 ms ~471 ms | |
- | FLUX.1 Canny Dev | 11.9B | INT4 Q8 | 68 tok/s 39 tok/s | ~762 ms ~837 ms | |
- | FLUX.1 Depth Dev | 11.9B | INT4 Q8 | 68 tok/s 39 tok/s | ~762 ms ~837 ms | |
- | FLUX.1 Dev | 11.9B | INT4 Q8 | 68 tok/s 39 tok/s | ~762 ms ~837 ms | |
- | FLUX.1 Fill Dev | 11.9B | INT4 Q8 | 68 tok/s 39 tok/s | ~762 ms ~837 ms | |
- | FLUX.1 Schnell | 11.9B | INT4 Q8 | 68 tok/s 39 tok/s | ~762 ms ~837 ms | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 | 68 tok/s 48 tok/s | ~522 ms ~528 ms | |
- | Typhoon-2-8B | 8B | INT4 Q8 | 68 tok/s 48 tok/s | ~522 ms ~528 ms | |
- | HunyuanVideo 1.5 720P T2V | 13B | INT4 Q8 | 63 tok/s 36 tok/s | ~831 ms ~912 ms | |
- | GLM-4-9B | 9B | INT4 Q8 | 62 tok/s 41 tok/s | ~585 ms ~642 ms | |
- | GLM-4-9B-Chat | 9B | INT4 Q8 | 62 tok/s 41 tok/s | ~585 ms ~642 ms | |
- | GLM-4-9B-Chat-1M | 9B | INT4 Q8 | 62 tok/s 41 tok/s | ~585 ms ~642 ms | |
- | Yi-9B | 9B | INT4 Q8 | 62 tok/s 41 tok/s | ~585 ms ~642 ms | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 Q8 | 61 tok/s 41 tok/s | ~597 ms ~655 ms | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 Q8 | 61 tok/s 41 tok/s | ~597 ms ~655 ms | |
- | Wan 2.1 I2V 14B 720P | 14B | INT4 Q8 | 59 tok/s 32 tok/s | ~893 ms ~1.1s | |
- | Wan 2.1 T2V 14B | 14B | INT4 Q8 | 59 tok/s 32 tok/s | ~893 ms ~1.1s | |
- | Wan 2.2 T2V A14B | 14B | INT4 Q8 | 59 tok/s 32 tok/s | ~893 ms ~1.1s | |
- | FLUX.2 Dev | 15B | INT4 Q8 | 56 tok/s 30 tok/s | ~954 ms ~1.1s | |
- | GreenMind-14B-R1 | 14B | INT4 | 44 tok/s | ~899 ms | |
- | Arete 27B | 27B | INT4 | 30 tok/s | ~2.0s | |
- | AREX-2 | 27B | INT4 | 30 tok/s | ~2.0s | |
- | PPLX Decider v1 27B | 27B | INT4 | 30 tok/s | ~2.0s | |
- | PPLX Decider v1.1 27B | 27B | INT4 | 30 tok/s | ~2.0s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
14B Models
8-bit
8B Models
16-bit
3B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
14B Models
LoRA
8B Models
Full Parameter Training
1B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA RTX 4070 Ti SUPER (16GB).
Inference Sweet Spot
Optimal for 8B models across extended context (32k+). 14B models fit comfortably at Q4 precision with standard context.
Bandwidth & Speed Profile
With 672.3 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 149 tok/s on an 8B Q4 model and 18 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning