NVIDIA RTX 4070 Ti (12GB)
The NVIDIA RTX 4070 Ti (12GB) features 12 GB of dedicated VRAM and 504 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
12 GB
Usable Memory Ceiling
12 GB
Bandwidth
504.2 GB/s
Max Inference Class
14B Models (Q4)
TDP
285W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA RTX 4070 Ti (12GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA RTX 4070 Ti (12GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#112 | Phi-4 Reasoning Plus | 14B | INT4 | 31 tok/s | ~1.2s | |
#125 | Qwen3.5-9B | 9B | INT4 Q8 | 67 tok/s 39 tok/s | ~640 ms ~704 ms | |
#128 | K2 Horizon 7B | 7B | INT4 Q8 | 82 tok/s 51 tok/s | ~502 ms ~509 ms | |
#135 | Gemma 4 12B | 11.95B | INT4 | 37 tok/s | ~922 ms | |
#140 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 121 tok/s 80 tok/s 46 tok/s | ~297 ms ~301 ms ~310 ms | |
#144 | GLM-4V | 9B | INT4 | 49 tok/s | ~645 ms | |
#150 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 22 tok/s 13 tok/s 6 tok/s | ~2.0s ~2.0s ~2.3s | |
#158 | Granite 4.2 8B | 8B | INT4 Q8 | 74 tok/s 43 tok/s | ~570 ms ~627 ms | |
#159 | Gemma 4 E4B | 8B | INT4 Q8 | 74 tok/s 43 tok/s | ~570 ms ~627 ms | |
#166 | Phi-4 | 14B | INT4 | 46 tok/s | ~984 ms | |
#178 | OLMo 3 7B Think | 7B | INT4 Q8 | 59 tok/s 40 tok/s | ~507 ms ~557 ms | |
#179 | DeepSeek-R1 14B | 14B | INT4 | 48 tok/s | ~983 ms | |
#183 | Ministral 3 14B | 14B | INT4 | 31 tok/s | ~1.2s | |
#186 | Gemma 3 12B | 12B | INT4 | 53 tok/s | ~846 ms | |
#189 | Ministral 3 8B | 8B | INT4 Q8 | 53 tok/s 34 tok/s | ~575 ms ~683 ms | |
#191 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 80 tok/s | ~161 ms ~163 ms ~168 ms | |
#192 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 80 tok/s | ~161 ms ~163 ms ~168 ms | |
#194 | Qwen2.5-14B | 14B | INT4 | 46 tok/s | ~984 ms | |
#200 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 103 tok/s 66 tok/s 33 tok/s | ~372 ms ~377 ms ~455 ms | |
#202 | DeepSeek-R1 8B | 8B | INT4 Q8 | 76 tok/s 46 tok/s | ~570 ms ~578 ms | |
#204 | Qwen3-4B | 4B | INT4 Q8 FP16 | 121 tok/s 80 tok/s 46 tok/s | ~297 ms ~301 ms ~310 ms | |
#205 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 98 tok/s 76 tok/s 50 tok/s | ~232 ms ~235 ms ~242 ms | |
#207 | Gemma 2 9B | 9B | INT4 Q8 | 67 tok/s 39 tok/s | ~640 ms ~704 ms | |
#208 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 257 tok/s 211 tok/s 152 tok/s | ~69 ms ~70 ms ~72 ms | |
#209 | Qwen3-14B | 14B | INT4 | 46 tok/s | ~984 ms | |
#210 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 121 tok/s 80 tok/s 46 tok/s | ~297 ms ~301 ms ~310 ms | |
#211 | Qwen3-8B | 8B | INT4 Q8 | 74 tok/s 43 tok/s | ~570 ms ~627 ms | |
#213 | DeepSeek-R1 7B | 7B | INT4 Q8 | 84 tok/s 52 tok/s | ~502 ms ~509 ms | |
#214 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 191 tok/s 143 tok/s 91 tok/s | ~138 ms ~140 ms ~144 ms | |
#215 | Llama 3.1 8B | 8B | INT4 Q8 | 74 tok/s 43 tok/s | ~570 ms ~627 ms | |
#216 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 295 tok/s 274 tok/s 234 tok/s | ~28 ms ~28 ms ~29 ms | |
#218 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 279 tok/s 238 tok/s 178 tok/s | ~53 ms ~54 ms ~55 ms | |
#219 | Llama 3 8B | 8B | INT4 Q8 | 74 tok/s 43 tok/s | ~570 ms ~627 ms | |
#221 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 125 tok/s 83 tok/s 48 tok/s | ~283 ms ~287 ms ~296 ms | |
#223 | Qwen2.5-7B | 7B | INT4 Q8 | 82 tok/s 51 tok/s | ~502 ms ~509 ms | |
#224 | Ministral-8B-2410 | 8B | INT4 Q8 | 74 tok/s 43 tok/s | ~570 ms ~627 ms | |
#225 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~123 ms ~125 ms ~128 ms | |
#226 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 143 tok/s 99 tok/s 58 tok/s | ~229 ms ~232 ms ~239 ms | |
#228 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 125 tok/s 83 tok/s 48 tok/s | ~283 ms ~287 ms ~296 ms | |
#229 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 | 97 tok/s 83 tok/s | ~176 ms ~177 ms | |
#230 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 | 79 tok/s 49 tok/s | ~522 ms ~530 ms | |
#231 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 237 tok/s 190 tok/s 132 tok/s | ~86 ms ~87 ms ~90 ms | |
#232 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 80 tok/s | ~161 ms ~163 ms ~168 ms | |
#233 | Phi-3-medium | 14B | INT4 | 46 tok/s | ~984 ms | |
#234 | OLMo 3 7B Instruct | 7B | INT4 Q8 | 59 tok/s 40 tok/s | ~507 ms ~557 ms | |
#236 | Qwen2-7B | 7B | INT4 Q8 | 82 tok/s 51 tok/s | ~502 ms ~509 ms | |
#237 | Phi-3-small | 7B | INT4 Q8 | 82 tok/s 51 tok/s | ~502 ms ~509 ms | |
#238 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 237 tok/s 190 tok/s 132 tok/s | ~86 ms ~87 ms ~90 ms | |
#239 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 | 79 tok/s 49 tok/s | ~522 ms ~530 ms | |
#241 | Gemma 1 7B | 7B | INT4 Q8 | 59 tok/s 40 tok/s | ~507 ms ~557 ms | |
#242 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 187 tok/s 134 tok/s 82 tok/s | ~160 ms ~163 ms ~167 ms | |
#243 | ChatGLM3-6B | 6B | INT4 Q8 | 65 tok/s 47 tok/s | ~438 ms ~444 ms | |
#244 | ChatGLM2-6B | 6B | INT4 Q8 | 65 tok/s 47 tok/s | ~438 ms ~444 ms | |
#245 | ChatGLM-6B | 6B | INT4 Q8 | 65 tok/s 47 tok/s | ~438 ms ~444 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 319 tok/s 290 tok/s 241 tok/s | ~30 ms ~30 ms ~31 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 292 tok/s 253 tok/s 195 tok/s | ~45 ms ~46 ms ~47 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 292 tok/s 253 tok/s 195 tok/s | ~45 ms ~46 ms ~47 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 287 tok/s 265 tok/s 224 tok/s | ~30 ms ~31 ms ~31 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 287 tok/s 265 tok/s 224 tok/s | ~30 ms ~31 ms ~31 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 279 tok/s 238 tok/s 178 tok/s | ~53 ms ~54 ms ~55 ms | |
- | PixArt-α XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 278 tok/s 236 tok/s 177 tok/s | ~54 ms ~55 ms ~56 ms | |
- | PixArt-Σ XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 278 tok/s 236 tok/s 177 tok/s | ~54 ms ~55 ms ~56 ms | |
- | Nemotron 3.5 Parse | 0.63B | INT4 Q8 FP16 | 276 tok/s 233 tok/s 174 tok/s | ~56 ms ~56 ms ~58 ms | |
- | SD 2.1 Turbo | 0.86B | INT4 Q8 FP16 | 251 tok/s 204 tok/s 145 tok/s | ~74 ms ~75 ms ~77 ms | |
- | Stable Diffusion 1.4 | 0.86B | INT4 Q8 FP16 | 251 tok/s 204 tok/s 145 tok/s | ~74 ms ~75 ms ~77 ms | |
- | Stable Diffusion 2.1 | 0.86B | INT4 Q8 FP16 | 251 tok/s 204 tok/s 145 tok/s | ~74 ms ~75 ms ~77 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 248 tok/s 196 tok/s 135 tok/s | ~86 ms ~87 ms ~90 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 237 tok/s 190 tok/s 132 tok/s | ~86 ms ~87 ms ~90 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 237 tok/s 190 tok/s 132 tok/s | ~86 ms ~87 ms ~90 ms | |
- | Kandinsky 2.2 Decoder | 1.22B | INT4 Q8 FP16 | 220 tok/s 172 tok/s 115 tok/s | ~103 ms ~104 ms ~107 ms | |
- | Wan 2.1 T2V 1.3B | 1.3B | INT4 Q8 FP16 | 215 tok/s 166 tok/s 111 tok/s | ~108 ms ~110 ms ~113 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~123 ms ~125 ms ~128 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~123 ms ~125 ms ~128 ms | |
- | SVD Image-to-Video XT | 1.5B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~123 ms ~125 ms ~128 ms | |
- | SVD Image-to-Video XT 1.1 | 1.5B | INT4 Q8 FP16 | 203 tok/s 154 tok/s 100 tok/s | ~123 ms ~125 ms ~128 ms | |
- | LFM3 1.7B | 1.7B | INT4 Q8 FP16 | 191 tok/s 143 tok/s 91 tok/s | ~138 ms ~140 ms ~144 ms | |
- | LFM3 1.7B Base | 1.7B | INT4 Q8 FP16 | 191 tok/s 143 tok/s 91 tok/s | ~138 ms ~140 ms ~144 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 191 tok/s 143 tok/s 91 tok/s | ~138 ms ~140 ms ~144 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 186 tok/s 138 tok/s 87 tok/s | ~145 ms ~147 ms ~152 ms | |
- | CogVideoX 2B | 2B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 80 tok/s | ~161 ms ~163 ms ~168 ms | |
- | LFM3 Audio 2B | 2B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 80 tok/s | ~161 ms ~163 ms ~168 ms | |
- | LFM3 Omni 2B | 2B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 80 tok/s | ~161 ms ~163 ms ~168 ms | |
- | LFM3 VL 2B | 2B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 80 tok/s | ~161 ms ~163 ms ~168 ms | |
- | Liquid D1 2B | 2B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 80 tok/s | ~161 ms ~163 ms ~168 ms | |
- | LTX-Video 0.9.1 | 2B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 80 tok/s | ~161 ms ~163 ms ~168 ms | |
- | Moondream 3.1 | 9B (2B active) | INT4 Q8 | 177 tok/s 123 tok/s | ~180 ms ~197 ms | |
- | Stable Diffusion 3 Medium | 2B | INT4 Q8 FP16 | 177 tok/s 129 tok/s 80 tok/s | ~161 ms ~163 ms ~168 ms | |
- | Kandinsky 3.0 | 2.1B | INT4 Q8 FP16 | 173 tok/s 125 tok/s 78 tok/s | ~168 ms ~170 ms ~175 ms | |
- | LTX-2 | 2.2B | INT4 Q8 FP16 | 169 tok/s 121 tok/s 75 tok/s | ~174 ms ~177 ms ~182 ms | |
- | SDXL 1.0 Refiner | 2.31B | INT4 Q8 FP16 | 165 tok/s 118 tok/s 72 tok/s | ~182 ms ~184 ms ~190 ms | |
- | Stable Diffusion 3.5 Medium | 2.5B | INT4 Q8 FP16 | 158 tok/s 112 tok/s 68 tok/s | ~195 ms ~197 ms ~203 ms | |
- | SDXL 1.0 Base | 2.56B | INT4 Q8 FP16 | 156 tok/s 110 tok/s 66 tok/s | ~199 ms ~201 ms ~207 ms | |
- | SDXL Turbo | 2.56B | INT4 Q8 FP16 | 156 tok/s 110 tok/s 66 tok/s | ~199 ms ~201 ms ~207 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 151 tok/s 102 tok/s 60 tok/s | ~228 ms ~231 ms ~238 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 149 tok/s 126 tok/s 91 tok/s | ~110 ms ~112 ms ~115 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 149 tok/s 126 tok/s 91 tok/s | ~110 ms ~112 ms ~115 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 149 tok/s 126 tok/s 91 tok/s | ~110 ms ~112 ms ~115 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 | 146 tok/s | ~223 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 145 tok/s 99 tok/s 59 tok/s | ~228 ms ~232 ms ~239 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 143 tok/s 99 tok/s 58 tok/s | ~229 ms ~232 ms ~239 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 143 tok/s 99 tok/s 58 tok/s | ~229 ms ~232 ms ~239 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 143 tok/s 99 tok/s 58 tok/s | ~229 ms ~232 ms ~239 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 143 tok/s 99 tok/s 58 tok/s | ~229 ms ~232 ms ~239 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 143 tok/s 99 tok/s 58 tok/s | ~229 ms ~232 ms ~239 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 143 tok/s 99 tok/s 58 tok/s | ~229 ms ~232 ms ~239 ms | |
- | FLUX.2 Klein 4B | 4B | INT4 Q8 FP16 | 121 tok/s 80 tok/s 46 tok/s | ~297 ms ~301 ms ~310 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 121 tok/s 80 tok/s 46 tok/s | ~297 ms ~301 ms ~310 ms | |
- | Mage-VL Cowork | 4B | INT4 Q8 FP16 | 121 tok/s 80 tok/s 46 tok/s | ~297 ms ~301 ms ~310 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 121 tok/s 80 tok/s 46 tok/s | ~297 ms ~301 ms ~310 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 116 tok/s 95 tok/s 66 tok/s | ~164 ms ~166 ms ~170 ms | |
- | CogVideoX 1.5 5B | 5B | INT4 Q8 FP16 | 104 tok/s 67 tok/s 36 tok/s | ~365 ms ~371 ms ~414 ms | |
- | CogVideoX 5B | 5B | INT4 Q8 FP16 | 104 tok/s 67 tok/s 36 tok/s | ~365 ms ~371 ms ~414 ms | |
- | Wan 2.2 TI2V 5B | 5B | INT4 Q8 FP16 | 104 tok/s 67 tok/s 36 tok/s | ~365 ms ~371 ms ~414 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 102 tok/s 81 tok/s 54 tok/s | ~211 ms ~214 ms ~220 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 98 tok/s 76 tok/s 50 tok/s | ~232 ms ~235 ms ~242 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 98 tok/s 76 tok/s 50 tok/s | ~232 ms ~235 ms ~242 ms | |
- | Falcon-7B | 7B | INT4 Q8 | 85 tok/s 52 tok/s | ~501 ms ~509 ms | |
- | AuraFlow v0.2 | 6.8B | INT4 Q8 | 83 tok/s 52 tok/s | ~488 ms ~495 ms | |
- | AuraFlow v0.3 | 6.8B | INT4 Q8 | 83 tok/s 52 tok/s | ~488 ms ~495 ms | |
- | Falcon3-7B | 7B | INT4 Q8 | 82 tok/s 51 tok/s | ~502 ms ~509 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 | 82 tok/s 51 tok/s | ~502 ms ~509 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 | 79 tok/s 49 tok/s | ~522 ms ~530 ms | |
- | Youtu Parsing Omni | 8B | INT4 Q8 | 76 tok/s 46 tok/s | ~570 ms ~578 ms | |
- | Stable Diffusion 3.5 Large | 8.1B | INT4 Q8 | 73 tok/s 43 tok/s | ~577 ms ~635 ms | |
- | Stable Diffusion 3.5 Large Turbo | 8.1B | INT4 Q8 | 73 tok/s 43 tok/s | ~577 ms ~635 ms | |
- | FLUX.2 Klein 9B | 9B | INT4 Q8 | 67 tok/s 39 tok/s | ~640 ms ~704 ms | |
- | LensVLM 9B | 9B | INT4 Q8 | 67 tok/s 39 tok/s | ~640 ms ~704 ms | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 | 67 tok/s 39 tok/s | ~640 ms ~704 ms | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 | 65 tok/s 47 tok/s | ~438 ms ~444 ms | |
- | Yi-6B | 6B | INT4 Q8 | 65 tok/s 47 tok/s | ~438 ms ~444 ms | |
- | Falcon3-10B | 10B | INT4 Q8 | 62 tok/s 34 tok/s | ~708 ms ~842 ms | |
- | Mochi 1 Preview | 10B | INT4 Q8 | 62 tok/s 34 tok/s | ~708 ms ~842 ms | |
- | Falcon2-11B | 11B | INT4 Q8 | 59 tok/s 32 tok/s | ~776 ms ~923 ms | |
- | Hunyuan Lite | 7B | INT4 Q8 | 59 tok/s 40 tok/s | ~507 ms ~557 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 | 59 tok/s 40 tok/s | ~507 ms ~557 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 | 58 tok/s 39 tok/s | ~513 ms ~564 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 | 58 tok/s 39 tok/s | ~513 ms ~564 ms | |
- | FLUX.1 Canny Dev | 11.9B | INT4 | 53 tok/s | ~840 ms | |
- | FLUX.1 Depth Dev | 11.9B | INT4 | 53 tok/s | ~840 ms | |
- | FLUX.1 Dev | 11.9B | INT4 | 53 tok/s | ~840 ms | |
- | FLUX.1 Fill Dev | 11.9B | INT4 | 53 tok/s | ~840 ms | |
- | FLUX.1 Schnell | 11.9B | INT4 | 53 tok/s | ~840 ms | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 | 53 tok/s 34 tok/s | ~575 ms ~683 ms | |
- | Typhoon-2-8B | 8B | INT4 Q8 | 53 tok/s 34 tok/s | ~575 ms ~683 ms | |
- | GLM-4-9B | 9B | INT4 | 49 tok/s | ~645 ms | |
- | GLM-4-9B-Chat | 9B | INT4 | 49 tok/s | ~645 ms | |
- | GLM-4-9B-Chat-1M | 9B | INT4 | 49 tok/s | ~645 ms | |
- | HunyuanVideo 1.5 720P T2V | 13B | INT4 | 49 tok/s | ~915 ms | |
- | Yi-9B | 9B | INT4 | 49 tok/s | ~645 ms | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 | 48 tok/s | ~659 ms | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 | 48 tok/s | ~659 ms | |
- | Wan 2.1 I2V 14B 720P | 14B | INT4 | 46 tok/s | ~984 ms | |
- | Wan 2.1 T2V 14B | 14B | INT4 | 46 tok/s | ~984 ms | |
- | Wan 2.2 T2V A14B | 14B | INT4 | 46 tok/s | ~984 ms | |
- | FLUX.2 Dev | 15B | INT4 | 44 tok/s | ~1.1s | |
- | GreenMind-14B-R1 | 14B | INT4 | 31 tok/s | ~1.2s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
14B Models
8-bit
8B Models
16-bit
3B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
14B Models
LoRA
8B Models
Full Parameter Training
1B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA RTX 4070 Ti (12GB).
Inference Sweet Spot
Optimal for 8B models across extended context (32k+). 14B models fit comfortably at Q4 precision with standard context.
Bandwidth & Speed Profile
With 504.2 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 112 tok/s on an 8B Q4 model and 13 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning