ApX logoApX logo

Llama 4 Maverick

Active Parameters

400B

Context Length

1M

Modality

Multimodal

Auxiliary Parameters

850M

Architecture

Mixture of Experts

KV Cache Rate

480 KB/tok

License

Llama 4 Community License Agreement

Release Date

5 Apr 2025

Knowledge Cutoff

Aug 2024

API Pricing (per 1M)

Input: $0.63 · Output: $1.80

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 782.97 GB

FP16 · 8K context

Base Weights: 776.72 GB

KV Cache: 3.75 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

8x NVIDIA H200 141GB

783.0 / 1015.2 GB usable · TP 8

Determine requirements for your specific workload:

Upgrade to Pro

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.51.0.dev0

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 12.3k · Context: 1M · Vocab: 202kx 120 layersRMSNormPre-AttentionGrouped-Query Attention96Q / 8KV headsHead dim: 128+RMSNormPre-FFNSparse MoE FFN (2/128 experts)Swish+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#164

BenchmarkScoreRank

Professional Knowledge

MMLU Pro

0.805

30

0.319

31

Software Engineering

SWE-bench Verified

0.21

38

Graduate-Level QA

GPQA

0.698

85

Agentic Index

Artificial Analysis

0.6

124

0.16

136

General Text

Text Arena

1327

158

Intelligence Index

Artificial Analysis

0.09

257

StackEval

Archived
ProLLM Stack Eval

0.923

7

QA Assistant

Archived
ProLLM QA Assistant

0.949

10

General Knowledge

Reference
MMLU

0.855

10

Coding

Archived
Aider Coding

0.16

20

Summarization

Archived
ProLLM Summarization

0.72

20

Rankings

Overall Rank

#164

Coding Rank

#138

About Llama 4 Maverick

Llama 4 Maverick is Meta's natively multimodal Mixture-of-Experts model with 400B total parameters activating 17B per token for production inference. It provides low-latency image and text understanding with early fusion across long contexts.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

96

Key-Value Heads

8

Attention Head Dimension

128

Position Embedding

ROPE

RoPE Theta

500,000

Normalization

RMS Normalization

Activation Function

Swish

Dimensions

Base Precision

FP16

Auxiliary Parameters

850M

Hidden Dimension Size

12,288

Number of Layers

120

FFN Intermediate Size (Dense)

8,192

Tokenizer

Vocabulary Size

202,048

Mixture of Experts

Total Expert Parameters

17.0B

Number of Experts

128

Active Experts

2

About Llama 4

Meta's Llama 4 model family implements a Mixture-of-Experts (MoE) architecture for efficient scaling. It features native multimodality through early fusion of text, images, and video. This iteration also supports significantly extended context lengths, with models capable of processing up to 10 million tokens.


Other Llama 4 Models