What are the best models to run on Instinct MI300A (128GB)?

Based on model rankings and 128 GB usable VRAM:

Best Models to Run on Instinct MI300A (128GB) (October 2026)

Q: How are VRAM, throughput, and latency calculated?

Calculations are determined using apxml.com modeling, combining LLM architecture analysis with GPU hardware physics. VRAM and Context Scaling: Memory requirements are determined from model architecture, including parameter count, layer configuration, and attention structure. The KV cache footprint scales deterministically based on context length, batch size, and cache precision. Throughput (TPS): Token generation is calculated in the memory-bandwidth-bound decode phase, measuring the time required to stream model weights and KV cache relative to hardware memory bandwidth. Latency (TTFT): Time to first token is calculated from the compute operations required during prompt prefill relative to hardware tensor compute capacity.

Q: How accurate are these estimates?

Calculations are calibrated against empirical ground-truth benchmark runs across diverse architectures and hardware. Because the underlying mechanics are benchmark-calibrated, estimates remain accurate across different configurations without needing to run physical tests for every scenario. VRAM Accuracy: Memory estimates are deterministic and closely match actual allocation. Minor differences typically stay within 3% to 5% due to CUDA runtime overhead or memory allocator fragmentation. Speed and Latency Accuracy: Throughput and TTFT typically vary within 5% to 10% depending on the specific inference runtime, kernel implementations, and system power limits.

Inference Capacity

Maximum model parameter class runnable by weight precision.

4-bit

70B+ Models

8-bit

70B Models

16-bit

FP16

32B Models

Fine-Tuning Capacity

Hardware capacity for local training and adapter fine-tuning.

Context Length:

QLoRA

4-bit

70B+ Models

LoRA

16-bit

70B Models

Full Parameter Training

7B-8B Models

Workload Recommendations

Guidance for optimal precision and operational ceilings on Instinct MI300A (128GB).

Inference Sweet Spot

Suitable for 70B parameter models at Q4/Q8 with extended 32k context windows, or high-throughput batching of smaller dense architectures.

Bandwidth & Speed Profile

With 5300 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 1178 tok/s on an 8B Q4 model and 139 tok/s on a 70B Q4 model.

Frequently Asked Questions

How are VRAM, throughput, and latency calculated?

How accurate are these estimates?

Instinct MI300A (128GB)

VRAM Scaling by Context Length