Llama 4 Behemoth
Active Parameters
2T
Context Length
10M
Modality
Multimodal
Architecture
Mixture of Experts
KV Cache Rate
640 KB/tok
License
Llama 4 Community License Agreement
Release Date
-
Knowledge Cutoff
Aug 2024
API Pricing (per 1M)
Self-hosted only
Hardware & System Requirements
Precision
Context
Workload Memory: 4269.23 GB
FP16 · 8K context
Base Weights: 4261.73 GB
KV Cache: 5.00 GB
Runtime & Graphs: 2.50 GB
Minimum Required Hardware
Requires a multi-node cluster.
Determine requirements for your specific workload:
Open in VRAM Calculator
Architecture Diagram
Evaluation Benchmarks
No evaluation benchmarks for Llama 4 Behemoth available.
Rankings
Overall Rank
-
Coding Rank
-
About Llama 4 Behemoth
Llama 4 Behemoth is Meta's 2T-parameter multimodal Mixture-of-Experts research foundation model activating 288B parameters per token. Engineered with native early fusion, it serves as the high-capacity knowledge source for downstream distillation.
Specifications
Attention
Attention Structure
Grouped-Query Attention
Attention Heads
128
Key-Value Heads
8
Position Embedding
Absolute Position Embedding
Normalization
RMS Normalization
Activation Function
SwigLU
Dimensions
Base Precision
BF16
Hidden Dimension Size
16,384
Number of Layers
160
Mixture of Experts
Total Expert Parameters
288.0B
Number of Experts
16
Active Experts
2
Resources
About Llama 4
Meta's Llama 4 model family implements a Mixture-of-Experts (MoE) architecture for efficient scaling. It features native multimodality through early fusion of text, images, and video. This iteration also supports significantly extended context lengths, with models capable of processing up to 10 million tokens.