What are the best models to run on NVIDIA A800 (40GB)?

Based on model rankings and 40 GB usable VRAM:

Best Models to Run on NVIDIA A800 (40GB) (October 2026)

Q: How are VRAM, throughput, and latency calculated?

Calculations are determined using apxml.com modeling, combining LLM architecture analysis with GPU hardware physics. VRAM and Context Scaling: Memory requirements are determined from model architecture, including parameter count, layer configuration, and attention structure. The KV cache footprint scales deterministically based on context length, batch size, and cache precision. Throughput (TPS): Token generation is calculated in the memory-bandwidth-bound decode phase, measuring the time required to stream model weights and KV cache relative to hardware memory bandwidth. Latency (TTFT): Time to first token is calculated from the compute operations required during prompt prefill relative to hardware tensor compute capacity.

Q: How accurate are these estimates?

Calculations are calibrated against empirical ground-truth benchmark runs across diverse architectures and hardware. Because the underlying mechanics are benchmark-calibrated, estimates remain accurate across different configurations without needing to run physical tests for every scenario. VRAM Accuracy: Memory estimates are deterministic and closely match actual allocation. Minor differences typically stay within 3% to 5% due to CUDA runtime overhead or memory allocator fragmentation. Speed and Latency Accuracy: Throughput and TTFT typically vary within 5% to 10% depending on the specific inference runtime, kernel implementations, and system power limits.

Inference Capacity

Maximum model parameter class runnable by weight precision.

4-bit

32B Models

8-bit

32B Models

16-bit

FP16

14B Models

Fine-Tuning Capacity

Hardware capacity for local training and adapter fine-tuning.

Context Length:

QLoRA

4-bit

32B Models

LoRA

16-bit

14B Models

Full Parameter Training

3B Models

Workload Recommendations

Guidance for optimal precision and operational ceilings on NVIDIA A800 (40GB).

Inference Sweet Spot

Optimized for 14B models at uncompressed or Q8 precision, and 32B models at Q4 with up to 32k context.

Bandwidth & Speed Profile

With 1555.2 GB/s aggregate bandwidth, batch size 1 inference operates in a memory-bandwidth bound regime. Generates approximately 346 tok/s on an 8B Q4 model and 41 tok/s on a 70B Q4 model.

Frequently Asked Questions

How are VRAM, throughput, and latency calculated?

How accurate are these estimates?

NVIDIA A800 (40GB)

VRAM Scaling by Context Length