Active Parameters
400B
Context Length
1M
Modality
Multimodal
Auxiliary Parameters
850M
Architecture
Mixture of Experts
KV Cache Rate
480 KB/tok
License
Llama 4 Community License Agreement
Release Date
5 Apr 2025
Knowledge Cutoff
Aug 2024
API Pricing (per 1M)
Input: $0.63 · Output: $1.80
Precision
Context
Workload Memory: 782.97 GB
FP16 · 8K context
Base Weights: 776.72 GB
KV Cache: 3.75 GB
Runtime & Graphs: 2.50 GB
8x NVIDIA H200 141GB
783.0 / 1015.2 GB usable · TP 8
Determine requirements for your specific workload:
Upgrade to Pro
Checkpoint format
FP16
CUDA
>= 11.8
PyTorch
>= 2.1
Transformers
>= 4.51.0.dev0
| Architecture | Precision | Attention kernel |
|---|---|---|
Blackwell, Hopper | Native FP16 | FlashAttention-3 |
Ada Lovelace, Ampere | Native FP16 | FlashAttention-2 |
Rank
#164
| Benchmark | Score | Rank |
|---|---|---|
Professional Knowledge | 0.805 | 30 |
StackUnseen | 0.319 | 31 |
Software Engineering | 0.21 | 38 |
Graduate-Level QA | 0.698 | 85 |
Agentic Index | 0.6 | 124 |
Coding Index | 0.16 | 136 |
General Text | 1327 | 158 |
Intelligence Index | 0.09 | 257 |
StackEval Archived | 0.923 | 7 |
QA Assistant Archived | 0.949 | 10 |
General Knowledge Reference | 0.855 | 10 |
Coding Archived | 0.16 | 20 |
Summarization Archived | 0.72 | 20 |
Overall Rank
#164
Coding Rank
#138
Llama 4 Maverick is Meta's natively multimodal Mixture-of-Experts model with 400B total parameters activating 17B per token for production inference. It provides low-latency image and text understanding with early fusion across long contexts.
Attention
Attention Structure
Grouped-Query Attention
Attention Heads
96
Key-Value Heads
8
Attention Head Dimension
128
Position Embedding
ROPE
RoPE Theta
500,000
Normalization
RMS Normalization
Activation Function
Swish
Dimensions
Base Precision
FP16
Auxiliary Parameters
850M
Hidden Dimension Size
12,288
Number of Layers
120
FFN Intermediate Size (Dense)
8,192
Tokenizer
Vocabulary Size
202,048
Mixture of Experts
Total Expert Parameters
17.0B
Number of Experts
128
Active Experts
2
Meta's Llama 4 model family implements a Mixture-of-Experts (MoE) architecture for efficient scaling. It features native multimodality through early fusion of text, images, and video. This iteration also supports significantly extended context lengths, with models capable of processing up to 10 million tokens.