NVIDIA RTX 3080 Ti (12GB)
The NVIDIA RTX 3080 Ti (12GB) features 12 GB of dedicated VRAM and 912 GB/s memory bandwidth, supporting CUDA and TensorRT-LLM runtimes for full-precision and quantized inference.
Memory
12 GB
Usable Memory Ceiling
12 GB
Bandwidth
912.4 GB/s
Max Inference Class
14B Models (Q4)
TDP
350W
VRAM Scaling by Context Length
Estimated memory requirements as context length increases. Curves crossing above the reference line indicate Out of Memory (OOM).
Precision:
Best Models to Run on NVIDIA RTX 3080 Ti (12GB)
Inference throughput, memory compatibility, and generation speeds for open-weights LLMs on NVIDIA RTX 3080 Ti (12GB).
Sort by:
Context:
KV Cache:
GPUs:
| Rank | Model | Parameters | Precision | TPS (Approx.) | TTFT | Can Run |
|---|---|---|---|---|---|---|
#112 | Phi-4 Reasoning Plus | 14B | INT4 | 52 tok/s | ~1.3s | |
#125 | Qwen3.5-9B | 9B | INT4 Q8 | 106 tok/s 65 tok/s | ~742 ms ~811 ms | |
#128 | K2 Horizon 7B | 7B | INT4 Q8 | 125 tok/s 83 tok/s | ~582 ms ~586 ms | |
#135 | Gemma 4 12B | 11.95B | INT4 | 61 tok/s | ~1.1s | |
#140 | Qwen3.5-4B | 4B | INT4 Q8 FP16 | 173 tok/s 123 tok/s 75 tok/s | ~344 ms ~346 ms ~351 ms | |
#144 | GLM-4V | 9B | INT4 | 80 tok/s | ~746 ms | |
#150 | NVIDIA Nemotron 3 Nano 30B-A3B | 3.5B (30B active) | INT4 Q8 FP16 | 39 tok/s 23 tok/s 12 tok/s | ~2.3s ~2.3s ~2.6s | |
#158 | Granite 4.2 8B | 8B | INT4 Q8 | 115 tok/s 71 tok/s | ~662 ms ~723 ms | |
#159 | Gemma 4 E4B | 8B | INT4 Q8 | 115 tok/s 71 tok/s | ~662 ms ~723 ms | |
#166 | Phi-4 | 14B | INT4 | 76 tok/s | ~1.1s | |
#178 | OLMo 3 7B Think | 7B | INT4 Q8 | 94 tok/s 66 tok/s | ~585 ms ~639 ms | |
#179 | DeepSeek-R1 14B | 14B | INT4 | 78 tok/s | ~1.1s | |
#183 | Ministral 3 14B | 14B | INT4 | 52 tok/s | ~1.3s | |
#186 | Gemma 3 12B | 12B | INT4 | 86 tok/s | ~983 ms | |
#189 | Ministral 3 8B | 8B | INT4 Q8 | 87 tok/s 56 tok/s | ~664 ms ~784 ms | |
#191 | MiniCPM5-2B | 2B | INT4 Q8 FP16 | 231 tok/s 182 tok/s 124 tok/s | ~186 ms ~187 ms ~189 ms | |
#192 | Qwen3.5-2B | 2B | INT4 Q8 FP16 | 231 tok/s 182 tok/s 124 tok/s | ~186 ms ~187 ms ~189 ms | |
#194 | Qwen2.5-14B | 14B | INT4 | 76 tok/s | ~1.1s | |
#200 | Gemma 4 E2B | 5.1B | INT4 Q8 FP16 | 152 tok/s 104 tok/s 56 tok/s | ~431 ms ~434 ms ~516 ms | |
#202 | DeepSeek-R1 8B | 8B | INT4 Q8 | 117 tok/s 76 tok/s | ~661 ms ~666 ms | |
#204 | Qwen3-4B | 4B | INT4 Q8 FP16 | 173 tok/s 123 tok/s 75 tok/s | ~344 ms ~346 ms ~351 ms | |
#205 | Ministral 3 3B | 3B | INT4 Q8 FP16 | 146 tok/s 118 tok/s 81 tok/s | ~266 ms ~268 ms ~272 ms | |
#207 | Gemma 2 9B | 9B | INT4 Q8 | 106 tok/s 65 tok/s | ~742 ms ~811 ms | |
#208 | Qwen3.5-0.8B | 0.8B | INT4 Q8 FP16 | 298 tok/s 262 tok/s 206 tok/s | ~80 ms ~80 ms ~81 ms | |
#209 | Qwen3-14B | 14B | INT4 | 76 tok/s | ~1.1s | |
#210 | Gemma 3 4B | 4B | INT4 Q8 FP16 | 173 tok/s 123 tok/s 75 tok/s | ~344 ms ~346 ms ~351 ms | |
#211 | Qwen3-8B | 8B | INT4 Q8 | 115 tok/s 71 tok/s | ~662 ms ~723 ms | |
#213 | DeepSeek-R1 7B | 7B | INT4 Q8 | 128 tok/s 84 tok/s | ~582 ms ~586 ms | |
#214 | Qwen3-1.7B | 1.7B | INT4 Q8 FP16 | 244 tok/s 197 tok/s 137 tok/s | ~160 ms ~161 ms ~163 ms | |
#215 | Llama 3.1 8B | 8B | INT4 Q8 | 115 tok/s 71 tok/s | ~662 ms ~723 ms | |
#216 | Gemma 3 270M | 0.27B | INT4 Q8 FP16 | 325 tok/s 310 tok/s 280 tok/s | ~31 ms ~31 ms ~32 ms | |
#218 | Qwen3-0.6B | 0.6B | INT4 Q8 FP16 | 314 tok/s 283 tok/s 232 tok/s | ~61 ms ~61 ms ~62 ms | |
#219 | Llama 3 8B | 8B | INT4 Q8 | 115 tok/s 71 tok/s | ~662 ms ~723 ms | |
#221 | Phi-4-Mini | 3.8B | INT4 Q8 FP16 | 177 tok/s 127 tok/s 78 tok/s | ~328 ms ~330 ms ~335 ms | |
#223 | Qwen2.5-7B | 7B | INT4 Q8 | 125 tok/s 83 tok/s | ~582 ms ~586 ms | |
#224 | Ministral-8B-2410 | 8B | INT4 Q8 | 115 tok/s 71 tok/s | ~662 ms ~723 ms | |
#225 | DeepSeek-R1 1.5B | 1.5B | INT4 Q8 FP16 | 254 tok/s 208 tok/s 148 tok/s | ~142 ms ~143 ms ~145 ms | |
#226 | Llama 3.2 3B | 3B | INT4 Q8 FP16 | 198 tok/s 147 tok/s 94 tok/s | ~265 ms ~266 ms ~270 ms | |
#228 | Phi-3-mini | 3.8B | INT4 Q8 FP16 | 177 tok/s 127 tok/s 78 tok/s | ~328 ms ~330 ms ~335 ms | |
#229 | Gemma 3n E2B IT | 6B (2B active) | INT4 Q8 | 145 tok/s 128 tok/s | ~200 ms ~201 ms | |
#230 | Mistral-7B-Instruct-v0.1 | 7.3B | INT4 Q8 | 122 tok/s 80 tok/s | ~606 ms ~610 ms | |
#231 | Llama 3.2 1B | 1B | INT4 Q8 FP16 | 282 tok/s 243 tok/s 185 tok/s | ~99 ms ~100 ms ~101 ms | |
#232 | Gemma 2 2B | 2B | INT4 Q8 FP16 | 231 tok/s 182 tok/s 124 tok/s | ~186 ms ~187 ms ~189 ms | |
#233 | Phi-3-medium | 14B | INT4 | 76 tok/s | ~1.1s | |
#234 | OLMo 3 7B Instruct | 7B | INT4 Q8 | 94 tok/s 66 tok/s | ~585 ms ~639 ms | |
#236 | Qwen2-7B | 7B | INT4 Q8 | 125 tok/s 83 tok/s | ~582 ms ~586 ms | |
#237 | Phi-3-small | 7B | INT4 Q8 | 125 tok/s 83 tok/s | ~582 ms ~586 ms | |
#238 | Gemma 3 1B | 1B | INT4 Q8 FP16 | 282 tok/s 243 tok/s 185 tok/s | ~99 ms ~100 ms ~101 ms | |
#239 | Mistral-7B-Instruct-v0.2 | 7.3B | INT4 Q8 | 122 tok/s 80 tok/s | ~606 ms ~610 ms | |
#241 | Gemma 1 7B | 7B | INT4 Q8 | 94 tok/s 66 tok/s | ~585 ms ~639 ms | |
#242 | Gemma 1 2B | 2B | INT4 Q8 FP16 | 241 tok/s 188 tok/s 126 tok/s | ~186 ms ~187 ms ~189 ms | |
#243 | ChatGLM3-6B | 6B | INT4 Q8 | 103 tok/s 77 tok/s | ~505 ms ~509 ms | |
#244 | ChatGLM2-6B | 6B | INT4 Q8 | 103 tok/s 77 tok/s | ~505 ms ~509 ms | |
#245 | ChatGLM-6B | 6B | INT4 Q8 | 103 tok/s 77 tok/s | ~505 ms ~509 ms | |
Unranked Models (Sorted by Speed) | ||||||
- | DINOv3 ViT-L/16 | 0.3B | INT4 Q8 FP16 | 340 tok/s 321 tok/s 286 tok/s | ~34 ms ~34 ms ~34 ms | |
- | Qwen2-0.5B | 0.5B | INT4 Q8 FP16 | 322 tok/s 295 tok/s 248 tok/s | ~51 ms ~52 ms ~52 ms | |
- | Qwen2.5-0.5B | 0.5B | INT4 Q8 FP16 | 322 tok/s 295 tok/s 248 tok/s | ~51 ms ~52 ms ~52 ms | |
- | ERNIE-4.5-0.3B | 0.3B | INT4 Q8 FP16 | 319 tok/s 304 tok/s 272 tok/s | ~34 ms ~34 ms ~34 ms | |
- | ERNIE-4.5-0.3B-Base | 0.3B | INT4 Q8 FP16 | 319 tok/s 304 tok/s 272 tok/s | ~34 ms ~34 ms ~34 ms | |
- | Liquid D1-Omni 600M | 0.6B | INT4 Q8 FP16 | 314 tok/s 283 tok/s 232 tok/s | ~61 ms ~61 ms ~62 ms | |
- | PixArt-α XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 313 tok/s 282 tok/s 231 tok/s | ~61 ms ~62 ms ~63 ms | |
- | PixArt-Σ XL 2 1024-MS | 0.61B | INT4 Q8 FP16 | 313 tok/s 282 tok/s 231 tok/s | ~61 ms ~62 ms ~63 ms | |
- | Nemotron 3.5 Parse | 0.63B | INT4 Q8 FP16 | 311 tok/s 279 tok/s 228 tok/s | ~64 ms ~64 ms ~65 ms | |
- | SD 2.1 Turbo | 0.86B | INT4 Q8 FP16 | 293 tok/s 256 tok/s 199 tok/s | ~85 ms ~86 ms ~87 ms | |
- | Stable Diffusion 1.4 | 0.86B | INT4 Q8 FP16 | 293 tok/s 256 tok/s 199 tok/s | ~85 ms ~86 ms ~87 ms | |
- | Stable Diffusion 2.1 | 0.86B | INT4 Q8 FP16 | 293 tok/s 256 tok/s 199 tok/s | ~85 ms ~86 ms ~87 ms | |
- | Falcon-1B | 1B | INT4 Q8 FP16 | 291 tok/s 249 tok/s 188 tok/s | ~99 ms ~100 ms ~101 ms | |
- | Falcon3-1B | 1B | INT4 Q8 FP16 | 282 tok/s 243 tok/s 185 tok/s | ~99 ms ~100 ms ~101 ms | |
- | MiniCPM-V 4.7 1B | 1B | INT4 Q8 FP16 | 282 tok/s 243 tok/s 185 tok/s | ~99 ms ~100 ms ~101 ms | |
- | Kandinsky 2.2 Decoder | 1.22B | INT4 Q8 FP16 | 269 tok/s 226 tok/s 167 tok/s | ~118 ms ~119 ms ~120 ms | |
- | Wan 2.1 T2V 1.3B | 1.3B | INT4 Q8 FP16 | 265 tok/s 221 tok/s 161 tok/s | ~125 ms ~125 ms ~127 ms | |
- | Qwen2-1.5B | 1.5B | INT4 Q8 FP16 | 254 tok/s 208 tok/s 148 tok/s | ~142 ms ~143 ms ~145 ms | |
- | Qwen2.5-1.5B | 1.5B | INT4 Q8 FP16 | 254 tok/s 208 tok/s 148 tok/s | ~142 ms ~143 ms ~145 ms | |
- | SVD Image-to-Video XT | 1.5B | INT4 Q8 FP16 | 254 tok/s 208 tok/s 148 tok/s | ~142 ms ~143 ms ~145 ms | |
- | SVD Image-to-Video XT 1.1 | 1.5B | INT4 Q8 FP16 | 254 tok/s 208 tok/s 148 tok/s | ~142 ms ~143 ms ~145 ms | |
- | LFM3 1.7B | 1.7B | INT4 Q8 FP16 | 244 tok/s 197 tok/s 137 tok/s | ~160 ms ~161 ms ~163 ms | |
- | LFM3 1.7B Base | 1.7B | INT4 Q8 FP16 | 244 tok/s 197 tok/s 137 tok/s | ~160 ms ~161 ms ~163 ms | |
- | Spark X2.5 1.7B | 1.7B | INT4 Q8 FP16 | 244 tok/s 197 tok/s 137 tok/s | ~160 ms ~161 ms ~163 ms | |
- | Hy-MT2 1.8B | 1.8B | INT4 Q8 FP16 | 240 tok/s 192 tok/s 133 tok/s | ~168 ms ~169 ms ~171 ms | |
- | CogVideoX 2B | 2B | INT4 Q8 FP16 | 231 tok/s 182 tok/s 124 tok/s | ~186 ms ~187 ms ~189 ms | |
- | LFM3 Audio 2B | 2B | INT4 Q8 FP16 | 231 tok/s 182 tok/s 124 tok/s | ~186 ms ~187 ms ~189 ms | |
- | LFM3 Omni 2B | 2B | INT4 Q8 FP16 | 231 tok/s 182 tok/s 124 tok/s | ~186 ms ~187 ms ~189 ms | |
- | LFM3 VL 2B | 2B | INT4 Q8 FP16 | 231 tok/s 182 tok/s 124 tok/s | ~186 ms ~187 ms ~189 ms | |
- | Liquid D1 2B | 2B | INT4 Q8 FP16 | 231 tok/s 182 tok/s 124 tok/s | ~186 ms ~187 ms ~189 ms | |
- | LTX-Video 0.9.1 | 2B | INT4 Q8 FP16 | 231 tok/s 182 tok/s 124 tok/s | ~186 ms ~187 ms ~189 ms | |
- | Moondream 3.1 | 9B (2B active) | INT4 Q8 | 231 tok/s 173 tok/s | ~208 ms ~227 ms | |
- | Stable Diffusion 3 Medium | 2B | INT4 Q8 FP16 | 231 tok/s 182 tok/s 124 tok/s | ~186 ms ~187 ms ~189 ms | |
- | Kandinsky 3.0 | 2.1B | INT4 Q8 FP16 | 227 tok/s 178 tok/s 120 tok/s | ~194 ms ~195 ms ~197 ms | |
- | LTX-2 | 2.2B | INT4 Q8 FP16 | 223 tok/s 174 tok/s 116 tok/s | ~201 ms ~203 ms ~206 ms | |
- | SDXL 1.0 Refiner | 2.31B | INT4 Q8 FP16 | 219 tok/s 169 tok/s 113 tok/s | ~210 ms ~211 ms ~214 ms | |
- | Stable Diffusion 3.5 Medium | 2.5B | INT4 Q8 FP16 | 213 tok/s 163 tok/s 107 tok/s | ~225 ms ~227 ms ~230 ms | |
- | SDXL 1.0 Base | 2.56B | INT4 Q8 FP16 | 211 tok/s 161 tok/s 105 tok/s | ~230 ms ~231 ms ~235 ms | |
- | SDXL Turbo | 2.56B | INT4 Q8 FP16 | 211 tok/s 161 tok/s 105 tok/s | ~230 ms ~231 ms ~235 ms | |
- | Falcon-3B | 3B | INT4 Q8 FP16 | 206 tok/s 151 tok/s 95 tok/s | ~264 ms ~266 ms ~270 ms | |
- | CroissantLLM Base | 1.3B | INT4 Q8 FP16 | 203 tok/s 179 tok/s 138 tok/s | ~126 ms ~126 ms ~128 ms | |
- | Phi-1 | 1.3B | INT4 Q8 FP16 | 203 tok/s 179 tok/s 138 tok/s | ~126 ms ~126 ms ~128 ms | |
- | Phi-1.5 | 1.3B | INT4 Q8 FP16 | 203 tok/s 179 tok/s 138 tok/s | ~126 ms ~126 ms ~128 ms | |
- | Mellum 2.1 12B A2.5B Thinking | 12B (2.5B active) | INT4 | 200 tok/s | ~258 ms | |
- | DeepSeek-R1 3B | 3B | INT4 Q8 FP16 | 199 tok/s 148 tok/s 94 tok/s | ~264 ms ~266 ms ~270 ms | |
- | Falcon3-3B | 3B | INT4 Q8 FP16 | 198 tok/s 147 tok/s 94 tok/s | ~265 ms ~266 ms ~270 ms | |
- | LFM2-VL 3B | 3B | INT4 Q8 FP16 | 198 tok/s 147 tok/s 94 tok/s | ~265 ms ~266 ms ~270 ms | |
- | Liquid D1 3B | 3B | INT4 Q8 FP16 | 198 tok/s 147 tok/s 94 tok/s | ~265 ms ~266 ms ~270 ms | |
- | Liquid D1-VL 3B | 3B | INT4 Q8 FP16 | 198 tok/s 147 tok/s 94 tok/s | ~265 ms ~266 ms ~270 ms | |
- | Liquid LFM2-VL 3B D1 | 3B | INT4 Q8 FP16 | 198 tok/s 147 tok/s 94 tok/s | ~265 ms ~266 ms ~270 ms | |
- | Qwen2.5-3B | 3B | INT4 Q8 FP16 | 198 tok/s 147 tok/s 94 tok/s | ~265 ms ~266 ms ~270 ms | |
- | FLUX.2 Klein 4B | 4B | INT4 Q8 FP16 | 173 tok/s 123 tok/s 75 tok/s | ~344 ms ~346 ms ~351 ms | |
- | FrogNano 4B | 4B | INT4 Q8 FP16 | 173 tok/s 123 tok/s 75 tok/s | ~344 ms ~346 ms ~351 ms | |
- | Mage-VL Cowork | 4B | INT4 Q8 FP16 | 173 tok/s 123 tok/s 75 tok/s | ~344 ms ~346 ms ~351 ms | |
- | Spark X2.5 4B | 4B | INT4 Q8 FP16 | 173 tok/s 123 tok/s 75 tok/s | ~344 ms ~346 ms ~351 ms | |
- | EmbeddingGemma 2 | 2B | INT4 Q8 FP16 | 167 tok/s 143 tok/s 104 tok/s | ~187 ms ~188 ms ~191 ms | |
- | CogVideoX 1.5 5B | 5B | INT4 Q8 FP16 | 153 tok/s 106 tok/s 60 tok/s | ~423 ms ~426 ms ~469 ms | |
- | CogVideoX 5B | 5B | INT4 Q8 FP16 | 153 tok/s 106 tok/s 60 tok/s | ~423 ms ~426 ms ~469 ms | |
- | Wan 2.2 TI2V 5B | 5B | INT4 Q8 FP16 | 153 tok/s 106 tok/s 60 tok/s | ~423 ms ~426 ms ~469 ms | |
- | Phi-2 | 2.7B | INT4 Q8 FP16 | 151 tok/s 124 tok/s 87 tok/s | ~243 ms ~244 ms ~247 ms | |
- | MaLLaM-3B | 3B | INT4 Q8 FP16 | 146 tok/s 118 tok/s 81 tok/s | ~266 ms ~268 ms ~272 ms | |
- | SmolLM3 3B | 3B | INT4 Q8 FP16 | 146 tok/s 118 tok/s 81 tok/s | ~266 ms ~268 ms ~272 ms | |
- | Falcon-7B | 7B | INT4 Q8 | 130 tok/s 85 tok/s | ~582 ms ~586 ms | |
- | AuraFlow v0.2 | 6.8B | INT4 Q8 | 128 tok/s 85 tok/s | ~566 ms ~570 ms | |
- | AuraFlow v0.3 | 6.8B | INT4 Q8 | 128 tok/s 85 tok/s | ~566 ms ~570 ms | |
- | Falcon3-7B | 7B | INT4 Q8 | 125 tok/s 83 tok/s | ~582 ms ~586 ms | |
- | Hy-MT2 7B | 7B | INT4 Q8 | 125 tok/s 83 tok/s | ~582 ms ~586 ms | |
- | Mistral-7B-v0.1 | 7.3B | INT4 Q8 | 122 tok/s 80 tok/s | ~606 ms ~610 ms | |
- | Youtu Parsing Omni | 8B | INT4 Q8 | 117 tok/s 76 tok/s | ~661 ms ~666 ms | |
- | Stable Diffusion 3.5 Large | 8.1B | INT4 Q8 | 114 tok/s 70 tok/s | ~669 ms ~731 ms | |
- | Stable Diffusion 3.5 Large Turbo | 8.1B | INT4 Q8 | 114 tok/s 70 tok/s | ~669 ms ~731 ms | |
- | FLUX.2 Klein 9B | 9B | INT4 Q8 | 106 tok/s 65 tok/s | ~742 ms ~811 ms | |
- | LensVLM 9B | 9B | INT4 Q8 | 106 tok/s 65 tok/s | ~742 ms ~811 ms | |
- | MiMo V2.6 Distill Qwen 9B | 9B | INT4 Q8 | 106 tok/s 65 tok/s | ~742 ms ~811 ms | |
- | ChatGLM3-6B-32K | 6B | INT4 Q8 | 103 tok/s 77 tok/s | ~505 ms ~509 ms | |
- | Yi-6B | 6B | INT4 Q8 | 103 tok/s 77 tok/s | ~505 ms ~509 ms | |
- | Falcon3-10B | 10B | INT4 Q8 | 98 tok/s 56 tok/s | ~822 ms ~971 ms | |
- | Mochi 1 Preview | 10B | INT4 Q8 | 98 tok/s 56 tok/s | ~822 ms ~971 ms | |
- | Falcon2-11B | 11B | INT4 Q8 | 95 tok/s 53 tok/s | ~901 ms ~1.1s | |
- | Hunyuan Lite | 7B | INT4 Q8 | 94 tok/s 66 tok/s | ~585 ms ~639 ms | |
- | OLMo 3 7B Base | 7B | INT4 Q8 | 94 tok/s 66 tok/s | ~585 ms ~639 ms | |
- | SEA-LION-7B | 7.1B | INT4 Q8 | 93 tok/s 65 tok/s | ~593 ms ~647 ms | |
- | SEA-LION-7B-Instruct | 7.1B | INT4 Q8 | 93 tok/s 65 tok/s | ~593 ms ~647 ms | |
- | Sahabat-AI-Llama3-8B-Instruct | 8B | INT4 Q8 | 87 tok/s 56 tok/s | ~664 ms ~784 ms | |
- | Typhoon-2-8B | 8B | INT4 Q8 | 87 tok/s 56 tok/s | ~664 ms ~784 ms | |
- | FLUX.1 Canny Dev | 11.9B | INT4 | 86 tok/s | ~975 ms | |
- | FLUX.1 Depth Dev | 11.9B | INT4 | 86 tok/s | ~975 ms | |
- | FLUX.1 Dev | 11.9B | INT4 | 86 tok/s | ~975 ms | |
- | FLUX.1 Fill Dev | 11.9B | INT4 | 86 tok/s | ~975 ms | |
- | FLUX.1 Schnell | 11.9B | INT4 | 86 tok/s | ~975 ms | |
- | HunyuanVideo 1.5 720P T2V | 13B | INT4 | 81 tok/s | ~1.1s | |
- | GLM-4-9B | 9B | INT4 | 80 tok/s | ~746 ms | |
- | GLM-4-9B-Chat | 9B | INT4 | 80 tok/s | ~746 ms | |
- | GLM-4-9B-Chat-1M | 9B | INT4 | 80 tok/s | ~746 ms | |
- | Yi-9B | 9B | INT4 | 80 tok/s | ~746 ms | |
- | Sahabat-AI-Gemma2-9B | 9.2B | INT4 | 78 tok/s | ~761 ms | |
- | Sahabat-AI-Gemma2-9B-Instruct | 9.2B | INT4 | 78 tok/s | ~761 ms | |
- | Wan 2.1 I2V 14B 720P | 14B | INT4 | 76 tok/s | ~1.1s | |
- | Wan 2.1 T2V 14B | 14B | INT4 | 76 tok/s | ~1.1s | |
- | Wan 2.2 T2V A14B | 14B | INT4 | 76 tok/s | ~1.1s | |
- | FLUX.2 Dev | 15B | INT4 | 72 tok/s | ~1.2s | |
- | GreenMind-14B-R1 | 14B | INT4 | 52 tok/s | ~1.3s | |
Inference Capacity
Maximum model parameter class runnable by weight precision.
Context:
4-bit
14B Models
8-bit
8B Models
16-bit
3B Models
Fine-Tuning Capacity
Hardware capacity for local training and adapter fine-tuning.
Context Length:
QLoRA
14B Models
LoRA
8B Models
Full Parameter Training
1B Models
Workload Recommendations
Guidance for optimal precision and operational ceilings on NVIDIA RTX 3080 Ti (12GB).
Inference Sweet Spot
Optimal for 8B models across extended context (32k+). 14B models fit comfortably at Q4 precision with standard context.
Bandwidth & Speed Profile
With 912.4 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 203 tok/s on an 8B Q4 model and 24 tok/s on a 70B Q4 model.
Frequently Asked Questions
©2025 ApX Machine Learning