How are VRAM, throughput, and latency calculated?

Calculations are determined using apxml.com modeling, combining LLM architecture analysis with GPU hardware physics. VRAM and Context Scaling: Memory requirements are determined from model architecture, including parameter count, layer configuration, and attention structure. The KV cache footprint scales deterministically based on context length, batch size, and cache precision. Throughput (TPS): Token generation is calculated in the memory-bandwidth-bound decode phase, measuring the time required to stream model weights and KV cache relative to hardware memory bandwidth. Latency (TTFT): Time to first token is calculated from the compute operations required during prompt prefill relative to hardware tensor compute capacity.

How accurate are these estimates?

Calculations are calibrated against empirical ground-truth benchmark runs across diverse architectures and hardware. Because the underlying mechanics are benchmark-calibrated, estimates remain accurate across different configurations without needing to run physical tests for every scenario. VRAM Accuracy: Memory estimates are deterministic and closely match actual allocation. Minor differences typically stay within 3% to 5% due to CUDA runtime overhead or memory allocator fragmentation. Speed and Latency Accuracy: Throughput and TTFT typically vary within 5% to 10% depending on the specific inference runtime, kernel implementations, and system power limits.

Best Models to Run on AMD Ryzen AI 9 HX 370 (96GB system, 72GB GPU-accessible) (October 2026)

Q: What are the best models to run on AMD Ryzen AI 9 HX 370 (96GB system, 72GB GPU-accessible)?

Based on model rankings and 72 GB usable VRAM: Top Overall: Qwen 3.8 27B (INT4) ~7 tok/s. Top Coding: Ling 3.0 Flash VL (INT4) ~27 tok/s. Optimal Daily Driver: Phi-4 Reasoning Plus (INT4) ~9 tok/s. Maximum Model Size: Ling 3.0 Flash (INT4) ~29 tok/s.

Rank	Model	Parameters	Precision	TPS (Approx.)	TTFT
#33	Qwen 3.8 27B	27B	INT4 Q8 FP16	7 tok/s 4 tok/s 2 tok/s	~24.6s ~24.7s ~24.9s
#48	Agnes 3.0 Flash	33B	INT4 Q8 FP16	6 tok/s 3 tok/s 2 tok/s	~29.9s ~30.0s ~33.0s
#61	Sarvam-105B	106B (10.3B active)	INT4	17 tok/s	~12.6s
#72	K2 Horizon MoVA 36B A4B	36B (4B active)	INT4 Q8 FP16	33 tok/s 21 tok/s 11 tok/s	~4.8s ~4.8s ~5.7s
#75	Ling 3.0 Flash VL	124B (5.5B active)	INT4	27 tok/s	~9.7s
#83	Qwen3.5-27B	27B	INT4 Q8 FP16	7 tok/s 4 tok/s 2 tok/s	~24.6s ~24.7s ~24.9s
#84	Qwen3.5-122B-A10B	122B (10B active)	INT4	12 tok/s	~14.0s
#92	Sarvam-30B	32B (2.4B active)	INT4 Q8 FP16	46 tok/s 32 tok/s 18 tok/s	~3.3s ~3.4s ~3.7s
#94	Qwen3.6 35B A3B	35B (3B active)	INT4 Q8 FP16	40 tok/s 27 tok/s 14 tok/s	~3.9s ~3.9s ~4.6s
#96	Muse Glimmer 30B	30B	INT4 Q8 FP16	6 tok/s 4 tok/s 2 tok/s	~27.3s ~27.4s ~30.0s
#100	Phi-4 Reasoning Plus	14B	INT4 Q8 FP16	9 tok/s 6 tok/s 3 tok/s	~12.9s ~13.0s ~13.1s
#101	MiMo V2 Flash	15B (309B active)	INT4 Q8 FP16	1 tok/s 1 tok/s 1 tok/s	~268.9s ~270.2s ~273.0s
#106	GLM-4.5-Air	106B (12B active)	INT4	4 tok/s	~16.7s
#108	Gemma 4 26B A4B	25.2B (3.8B active)	INT4 Q8 FP16	36 tok/s 23 tok/s 13 tok/s	~4.4s ~4.4s ~4.4s
#109	Gemma 4 12B	11.95B	INT4 Q8 FP16	11 tok/s 7 tok/s 4 tok/s	~11.1s ~11.1s ~11.2s
#113	Qwen3.5-9B	9B	INT4 Q8 FP16	20 tok/s 11 tok/s 6 tok/s	~8.4s ~8.4s ~8.5s
#114	Qwen3.5-35B-A3B	35B (3B active)	INT4 Q8 FP16	40 tok/s 27 tok/s 14 tok/s	~3.9s ~3.9s ~4.6s
#114	Ling 3.0 Flash	124B (5.1B active)	INT4	29 tok/s	~9.3s
#118	K2 Horizon 7B	7B	INT4 Q8 FP16	25 tok/s 14 tok/s 7 tok/s	~6.5s ~6.6s ~6.6s
#126	Qwen3 Next 80B A3B	80B (79B active)	INT4	2 tok/s	~71.3s
#128	NVIDIA Nemotron 3 Nano 30B-A3B	3.5B (30B active)	INT4 Q8 FP16	6 tok/s 3 tok/s 2 tok/s	~26.4s ~26.5s ~26.8s
#129	Gemma 4 31B	30.7B	INT4 Q8 FP16	6 tok/s 3 tok/s 2 tok/s	~27.9s ~28.0s ~30.7s
#131	Nemotron 3.5 Lightning	30B (3B active)	INT4 Q8 FP16	14 tok/s 12 tok/s 9 tok/s	~3.9s ~3.9s ~4.3s
#132	Qwen3.5-4B	4B	INT4 Q8 FP16	40 tok/s 24 tok/s 13 tok/s	~3.8s ~3.9s ~3.9s
#133	Qwen3-30B-A3B	30B (3B active)	INT4 Q8 FP16	40 tok/s 27 tok/s 15 tok/s	~3.8s ~3.8s ~4.2s
#136	GPT-OSS 20B	21B (3.6B active)	INT4 Q8 FP16	15 tok/s 13 tok/s 9 tok/s	~4.1s ~4.2s ~4.2s
#146	Granite 4.2 8B	8B	INT4 Q8 FP16	22 tok/s 13 tok/s 7 tok/s	~7.4s ~7.5s ~7.6s
#150	Gemma 4 E4B	8B	INT4 Q8 FP16	22 tok/s 13 tok/s 7 tok/s	~7.4s ~7.5s ~7.6s
#151	Nemotron 3.5 Lightning 30B A3B	30B (3B active)	INT4 Q8 FP16	40 tok/s 27 tok/s 15 tok/s	~3.8s ~3.8s ~4.2s
#153	Qwen3-32B	32B	INT4 Q8 FP16	6 tok/s 3 tok/s 2 tok/s	~29.0s ~29.2s ~32.0s
#154	Qwen2.5-72B	72B	INT4 Q8	3 tok/s 1 tok/s	~64.9s ~76.6s
#155	Mistral Large 3	41B (675B active)	INT4 Q8	1 tok/s 1 tok/s	~587.0s ~589.8s
#156	GLM-4	32B	INT4 Q8 FP16	5 tok/s 3 tok/s 1 tok/s	~29.1s ~29.2s ~34.7s
#157	DeepSeek-R1 70B	70B	INT4 Q8	3 tok/s 1 tok/s	~63.0s ~74.5s
#158	GLM-4V	9B	INT4 Q8 FP16	14 tok/s 9 tok/s 5 tok/s	~8.4s ~8.4s ~8.5s
#159	Gemma 3 27B	27B	INT4 Q8 FP16	7 tok/s 4 tok/s 2 tok/s	~24.6s ~24.7s ~24.9s
#160	Phi-4	14B	INT4 Q8 FP16	13 tok/s 7 tok/s 4 tok/s	~12.9s ~13.0s ~13.1s
#161	Llama 3.3 70B	70B	INT4 Q8	3 tok/s 1 tok/s	~63.1s ~74.5s
#162	DeepSeek-R1 32B	32B	INT4 Q8 FP16	6 tok/s 3 tok/s 2 tok/s	~29.0s ~29.2s ~32.0s
#163	Hunyuan Standard	52B (389B active)	INT4 Q8	1 tok/s 1 tok/s	~339.1s ~370.2s
#164	Qwen2.5-32B	32B	INT4 Q8 FP16	6 tok/s 3 tok/s 2 tok/s	~29.0s ~29.2s ~32.0s
#168	Magistral Small	24B	INT4 Q8 FP16	6 tok/s 4 tok/s 2 tok/s	~21.9s ~22.0s ~22.3s
#169	OLMo 3 7B Think	7B	INT4 Q8 FP16	17 tok/s 12 tok/s 7 tok/s	~6.6s ~6.6s ~6.7s
#170	OLMo 3.1 32B Instruct	32B	INT4 Q8 FP16	5 tok/s 3 tok/s 1 tok/s	~29.1s ~29.2s ~34.7s
#171	Llama 3.1 70B	70B	INT4 Q8	3 tok/s 1 tok/s	~63.1s ~74.5s
#172	DeepSeek-R1 14B	14B	INT4 Q8 FP16	13 tok/s 8 tok/s 4 tok/s	~12.9s ~13.0s ~13.1s
#173	Ministral 3 14B	14B	INT4 Q8 FP16	9 tok/s 6 tok/s 3 tok/s	~12.9s ~13.0s ~13.1s
#174	Mistral-Large-2407	123B	INT4	1 tok/s	~120.5s
#175	Llama 4 Scout	109B (17B active)	INT4	9 tok/s	~20.1s
#177	Gemma 3 12B	12B	INT4 Q8 FP16	15 tok/s 9 tok/s 4 tok/s	~11.1s ~11.2s ~11.3s
#178	Command A	111B	INT4	1 tok/s	~117.9s
#179	Qwen2-72B	72B	INT4 Q8	3 tok/s 1 tok/s	~64.9s ~76.6s
#180	OLMo 3 32B Base	32B	INT4 Q8 FP16	5 tok/s 3 tok/s 1 tok/s	~29.1s ~29.2s ~34.7s
#181	Mistral-Small-2501	24B	INT4 Q8 FP16	8 tok/s 4 tok/s 2 tok/s	~21.9s ~22.0s ~22.2s
#182	Ministral 3 8B	8B	INT4 Q8 FP16	15 tok/s 10 tok/s 6 tok/s	~7.5s ~7.5s ~7.6s
#183	Qwen3.5-2B	2B	INT4 Q8 FP16	70 tok/s 44 tok/s 24 tok/s	~2.0s ~2.0s ~2.1s
#184	MiniCPM5-2B	2B	INT4 Q8 FP16	70 tok/s 44 tok/s 24 tok/s	~2.0s ~2.0s ~2.1s
#185	Qwen2.5-14B	14B	INT4 Q8 FP16	13 tok/s 7 tok/s 4 tok/s	~12.9s ~13.0s ~13.1s
#186	Gemma 2 27B	27B	INT4 Q8 FP16	7 tok/s 4 tok/s 2 tok/s	~24.6s ~24.7s ~24.9s
#187	OLMo 3.1 32B Think	32B	INT4 Q8 FP16	5 tok/s 3 tok/s 1 tok/s	~29.1s ~29.2s ~34.7s
#188	Gemma 4 E2B	5.1B	INT4 Q8 FP16	33 tok/s 19 tok/s 10 tok/s	~4.8s ~4.8s ~4.9s
#189	Qwen3-4B	4B	INT4 Q8 FP16	40 tok/s 24 tok/s 13 tok/s	~3.8s ~3.9s ~3.9s
#190	DeepSeek-R1 8B	8B	INT4 Q8 FP16	23 tok/s 13 tok/s 7 tok/s	~7.4s ~7.5s ~7.6s
#191	Llama 3 70B	70B	INT4 Q8	3 tok/s 1 tok/s	~63.1s ~74.5s
#192	Command R Plus	104B	INT4	1 tok/s	~110.4s
#193	Ministral 3 3B	3B	INT4 Q8 FP16	31 tok/s 23 tok/s 14 tok/s	~2.9s ~3.0s ~3.0s
#195	Qwen3-14B	14B	INT4 Q8 FP16	13 tok/s 7 tok/s 4 tok/s	~12.9s ~13.0s ~13.1s
#196	Kimi Linear 48B A3B Instruct	48B (3B active)	INT4 Q8	11 tok/s 11 tok/s	~4.2s ~4.2s
#197	Gemma 2 9B	9B	INT4 Q8 FP16	20 tok/s 11 tok/s 6 tok/s	~8.4s ~8.4s ~8.5s
#198	Qwen2.5-7B	7B	INT4 Q8 FP16	25 tok/s 14 tok/s 7 tok/s	~6.5s ~6.6s ~6.6s
#199	Qwen3-8B	8B	INT4 Q8 FP16	22 tok/s 13 tok/s 7 tok/s	~7.4s ~7.5s ~7.6s
#200	Gemma 3 4B	4B	INT4 Q8 FP16	40 tok/s 24 tok/s 13 tok/s	~3.8s ~3.9s ~3.9s
#201	DeepSeek-R1 7B	7B	INT4 Q8 FP16	26 tok/s 15 tok/s 8 tok/s	~6.5s ~6.6s ~6.6s
#202	Llama 3.1 8B	8B	INT4 Q8 FP16	22 tok/s 13 tok/s 7 tok/s	~7.4s ~7.5s ~7.6s
#204	Qwen3-1.7B	1.7B	INT4 Q8 FP16	79 tok/s 51 tok/s 28 tok/s	~1.7s ~1.7s ~1.8s
#205	Command R	35B	INT4 Q8	4 tok/s 3 tok/s	~31.7s ~31.9s
#206	Gemma 3 270M	0.27B	INT4 Q8 FP16	185 tok/s 155 tok/s 112 tok/s	~273 ms ~274 ms ~276 ms
#208	Llama 3 8B	8B	INT4 Q8 FP16	22 tok/s 13 tok/s 7 tok/s	~7.4s ~7.5s ~7.6s
#209	Qwen3-0.6B	0.6B	INT4 Q8 FP16	162 tok/s 116 tok/s 71 tok/s	~610 ms ~612 ms ~618 ms
#210	Phi-4-Mini	3.8B	INT4 Q8 FP16	42 tok/s 25 tok/s 13 tok/s	~3.7s ~3.7s ~3.7s
#211	Mixtral-8x7B-v0.1	46.7B (7B active)	INT4 Q8	22 tok/s 13 tok/s	~7.6s ~7.6s
#212	Ministral-8B-2410	8B	INT4 Q8 FP16	22 tok/s 13 tok/s 7 tok/s	~7.4s ~7.5s ~7.6s
#213	DeepSeek-R1 1.5B	1.5B	INT4 Q8 FP16	87 tok/s 56 tok/s 32 tok/s	~1.5s ~1.5s ~1.6s
#214	Llama 3.2 3B	3B	INT4 Q8 FP16	51 tok/s 31 tok/s 17 tok/s	~2.9s ~2.9s ~3.0s
#215	Qwen3.5-0.8B	0.8B	INT4 Q8 FP16	135 tok/s 93 tok/s 55 tok/s	~825 ms ~829 ms ~836 ms
#216	Phi-3-mini	3.8B	INT4 Q8 FP16	42 tok/s 25 tok/s 13 tok/s	~3.7s ~3.7s ~3.7s
#217	Gemma 3n E2B IT	6B (2B active)	INT4 Q8 FP16	31 tok/s 25 tok/s 17 tok/s	~2.2s ~2.2s ~2.2s
#218	Mistral-7B-Instruct-v0.1	7.3B	INT4 Q8 FP16	24 tok/s 14 tok/s 7 tok/s	~6.8s ~6.8s ~6.9s
#219	Llama 3.2 1B	1B	INT4 Q8 FP16	115 tok/s 78 tok/s 45 tok/s	~1.1s ~1.1s ~1.1s
#220	Gemma 2 2B	2B	INT4 Q8 FP16	70 tok/s 44 tok/s 24 tok/s	~2.0s ~2.0s ~2.1s
#221	Phi-3-medium	14B	INT4 Q8 FP16	13 tok/s 7 tok/s 4 tok/s	~12.9s ~13.0s ~13.1s
#222	OLMo 3 7B Instruct	7B	INT4 Q8 FP16	17 tok/s 12 tok/s 7 tok/s	~6.6s ~6.6s ~6.7s
#223	Yi-34B	34B	INT4 Q8 FP16	4 tok/s 3 tok/s 1 tok/s	~30.8s ~31.0s ~36.8s
#224	Qwen2-7B	7B	INT4 Q8 FP16	25 tok/s 14 tok/s 7 tok/s	~6.5s ~6.6s ~6.6s
#225	Phi-3-small	7B	INT4 Q8 FP16	25 tok/s 14 tok/s 7 tok/s	~6.5s ~6.6s ~6.6s
#226	Gemma 3 1B	1B	INT4 Q8 FP16	115 tok/s 78 tok/s 45 tok/s	~1.1s ~1.1s ~1.1s
#227	Mistral-7B-Instruct-v0.2	7.3B	INT4 Q8 FP16	24 tok/s 14 tok/s 7 tok/s	~6.8s ~6.8s ~6.9s
#229	Gemma 1 7B	7B	INT4 Q8 FP16	17 tok/s 12 tok/s 7 tok/s	~6.6s ~6.6s ~6.7s
#230	Gemma 1 2B	2B	INT4 Q8 FP16	76 tok/s 47 tok/s 25 tok/s	~2.0s ~2.0s ~2.1s
#231	ChatGLM3-6B	6B	INT4 Q8 FP16	19 tok/s 13 tok/s 8 tok/s	~5.7s ~5.7s ~5.7s
#232	ChatGLM2-6B	6B	INT4 Q8 FP16	19 tok/s 13 tok/s 8 tok/s	~5.7s ~5.7s ~5.7s
#233	ChatGLM-6B	6B	INT4 Q8 FP16	19 tok/s 13 tok/s 8 tok/s	~5.7s ~5.7s ~5.7s
Unranked Models (Sorted by Speed)
-	Qwen2-0.5B	0.5B	INT4 Q8 FP16	180 tok/s 131 tok/s 82 tok/s	~505 ms ~507 ms ~512 ms
-	Qwen2.5-0.5B	0.5B	INT4 Q8 FP16	180 tok/s 131 tok/s 82 tok/s	~505 ms ~507 ms ~512 ms
-	ERNIE-4.5-0.3B	0.3B	INT4 Q8 FP16	173 tok/s 144 tok/s 103 tok/s	~304 ms ~305 ms ~308 ms
-	ERNIE-4.5-0.3B-Base	0.3B	INT4 Q8 FP16	173 tok/s 144 tok/s 103 tok/s	~304 ms ~305 ms ~308 ms
-	Falcon-1B	1B	INT4 Q8 FP16	126 tok/s 83 tok/s 47 tok/s	~1.0s ~1.1s ~1.1s
-	Falcon3-1B	1B	INT4 Q8 FP16	115 tok/s 78 tok/s 45 tok/s	~1.1s ~1.1s ~1.1s
-	Qwen2-1.5B	1.5B	INT4 Q8 FP16	87 tok/s 56 tok/s 32 tok/s	~1.5s ~1.5s ~1.6s
-	Qwen2.5-1.5B	1.5B	INT4 Q8 FP16	87 tok/s 56 tok/s 32 tok/s	~1.5s ~1.5s ~1.6s
-	Spark X2.5 1.7B	1.7B	INT4 Q8 FP16	79 tok/s 51 tok/s 28 tok/s	~1.7s ~1.7s ~1.8s
-	Hy-MT2 1.8B	1.8B	INT4 Q8 FP16	76 tok/s 48 tok/s 27 tok/s	~1.8s ~1.8s ~1.9s
-	Falcon-3B	3B	INT4 Q8 FP16	55 tok/s 33 tok/s 17 tok/s	~2.9s ~2.9s ~3.0s
-	CroissantLLM Base	1.3B	INT4 Q8 FP16	54 tok/s 43 tok/s 28 tok/s	~1.3s ~1.4s ~1.4s
-	Phi-1	1.3B	INT4 Q8 FP16	54 tok/s 43 tok/s 28 tok/s	~1.3s ~1.4s ~1.4s
-	Phi-1.5	1.3B	INT4 Q8 FP16	54 tok/s 43 tok/s 28 tok/s	~1.3s ~1.4s ~1.4s
-	DeepSeek-R1 3B	3B	INT4 Q8 FP16	52 tok/s 32 tok/s 17 tok/s	~2.9s ~2.9s ~3.0s
-	Falcon3-3B	3B	INT4 Q8 FP16	51 tok/s 31 tok/s 17 tok/s	~2.9s ~2.9s ~3.0s
-	Qwen2.5-3B	3B	INT4 Q8 FP16	51 tok/s 31 tok/s 17 tok/s	~2.9s ~2.9s ~3.0s
-	ERNIE-4.5-21B-A3B	21B (3B active)	INT4 Q8 FP16	43 tok/s 28 tok/s 16 tok/s	~3.6s ~3.6s ~3.6s
-	ERNIE-4.5-21B-A3B-Base	21B (3B active)	INT4 Q8 FP16	43 tok/s 28 tok/s 16 tok/s	~3.6s ~3.6s ~3.6s
-	ERNIE-4.5-VL-28B-A3B	28B (3B active)	INT4 Q8 FP16	41 tok/s 27 tok/s 15 tok/s	~3.8s ~3.8s ~4.1s
-	ERNIE-4.5-VL-28B-A3B-Base	28B (3B active)	INT4 Q8 FP16	41 tok/s 27 tok/s 15 tok/s	~3.8s ~3.8s ~4.1s
-	Hy-MT2 30B A3B	30B (3B active)	INT4 Q8 FP16	40 tok/s 27 tok/s 15 tok/s	~3.8s ~3.8s ~4.2s
-	Spark X2.5 4B	4B	INT4 Q8 FP16	40 tok/s 24 tok/s 13 tok/s	~3.8s ~3.9s ~3.9s
-	Phi-2	2.7B	INT4 Q8 FP16	33 tok/s 25 tok/s 15 tok/s	~2.7s ~2.7s ~2.7s
-	MaLLaM-3B	3B	INT4 Q8 FP16	31 tok/s 23 tok/s 14 tok/s	~2.9s ~3.0s ~3.0s
-	SmolLM3 3B	3B	INT4 Q8 FP16	31 tok/s 23 tok/s 14 tok/s	~2.9s ~3.0s ~3.0s
-	AliceAI Foundation 80B A3B Base	80B (3B active)	INT4	30 tok/s	~5.3s
-	Falcon-7B	7B	INT4 Q8 FP16	26 tok/s 15 tok/s 8 tok/s	~6.5s ~6.6s ~6.6s
-	Falcon3-7B	7B	INT4 Q8 FP16	25 tok/s 14 tok/s 7 tok/s	~6.5s ~6.6s ~6.6s
-	Hy-MT2 7B	7B	INT4 Q8 FP16	25 tok/s 14 tok/s 7 tok/s	~6.5s ~6.6s ~6.6s
-	Mistral-7B-v0.1	7.3B	INT4 Q8 FP16	24 tok/s 14 tok/s 7 tok/s	~6.8s ~6.8s ~6.9s
-	LensVLM 9B	9B	INT4 Q8 FP16	20 tok/s 11 tok/s 6 tok/s	~8.4s ~8.4s ~8.5s
-	MiMo V2.6 Distill Qwen 9B	9B	INT4 Q8 FP16	20 tok/s 11 tok/s 6 tok/s	~8.4s ~8.4s ~8.5s
-	ChatGLM3-6B-32K	6B	INT4 Q8 FP16	19 tok/s 13 tok/s 8 tok/s	~5.7s ~5.7s ~5.7s
-	Yi-6B	6B	INT4 Q8 FP16	19 tok/s 13 tok/s 8 tok/s	~5.7s ~5.7s ~5.7s
-	Falcon3-10B	10B	INT4 Q8 FP16	18 tok/s 10 tok/s 5 tok/s	~9.3s ~9.3s ~9.4s
-	Kimi-VL-A3B-Instruct	16B (3B active)	INT4 Q8 FP16	18 tok/s 15 tok/s 11 tok/s	~3.5s ~3.5s ~3.5s
-	Falcon2-11B	11B	INT4 Q8 FP16	17 tok/s 10 tok/s 5 tok/s	~10.2s ~10.2s ~10.3s
-	Hunyuan Lite	7B	INT4 Q8 FP16	17 tok/s 12 tok/s 7 tok/s	~6.6s ~6.6s ~6.7s
-	OLMo 3 7B Base	7B	INT4 Q8 FP16	17 tok/s 12 tok/s 7 tok/s	~6.6s ~6.6s ~6.7s
-	SEA-LION-7B	7.1B	INT4 Q8 FP16	17 tok/s 11 tok/s 7 tok/s	~6.7s ~6.7s ~6.7s
-	SEA-LION-7B-Instruct	7.1B	INT4 Q8 FP16	17 tok/s 11 tok/s 7 tok/s	~6.7s ~6.7s ~6.7s
-	Sahabat-AI-Llama3-8B-Instruct	8B	INT4 Q8 FP16	15 tok/s 10 tok/s 6 tok/s	~7.5s ~7.5s ~7.6s
-	Typhoon-2-8B	8B	INT4 Q8 FP16	15 tok/s 10 tok/s 6 tok/s	~7.5s ~7.5s ~7.6s
-	GLM-4-9B	9B	INT4 Q8 FP16	14 tok/s 9 tok/s 5 tok/s	~8.4s ~8.4s ~8.5s
-	GLM-4-9B-Chat	9B	INT4 Q8 FP16	14 tok/s 9 tok/s 5 tok/s	~8.4s ~8.4s ~8.5s
-	GLM-4-9B-Chat-1M	9B	INT4 Q8 FP16	14 tok/s 9 tok/s 5 tok/s	~8.4s ~8.4s ~8.5s
-	Laguna S 2.1	118B (8B active)	INT4	14 tok/s	~12.0s
-	Yi-9B	9B	INT4 Q8 FP16	14 tok/s 9 tok/s 5 tok/s	~8.4s ~8.4s ~8.5s
-	Sahabat-AI-Gemma2-9B	9.2B	INT4 Q8 FP16	13 tok/s 9 tok/s 5 tok/s	~8.6s ~8.6s ~8.7s
-	Sahabat-AI-Gemma2-9B-Instruct	9.2B	INT4 Q8 FP16	13 tok/s 9 tok/s 5 tok/s	~8.6s ~8.6s ~8.7s
-	GreenMind-14B-R1	14B	INT4 Q8 FP16	9 tok/s 6 tok/s 3 tok/s	~12.9s ~13.0s ~13.1s
-	Falcon-40B	40B	INT4 Q8	5 tok/s 3 tok/s	~36.1s ~36.3s
-	Hunyuan A13B	80B (13B active)	INT4	5 tok/s	~14.0s
-	Kimi-Dev-72B	72B	INT4	2 tok/s	~65.0s
-	Typhoon-2-70B	70B	INT4	2 tok/s	~63.1s

AMD Ryzen AI 9 HX 370 (96GB system, 72GB GPU-accessible)

VRAM Scaling by Context Length

Best Models to Run on AMD Ryzen AI 9 HX 370 (96GB system, 72GB GPU-accessible)

Inference Capacity

Fine-Tuning Capacity

Workload Recommendations

Frequently Asked Questions