Llama 4 Behemoth
Active Parameters
2T
Context Length
10M
Modality
Multimodal
Architecture
Mixture of Experts (MoE)
License
Llama 4 Community License Agreement
Release Date
-
Knowledge Cutoff
Aug 2024
System Requirements
VRAM requirements for different quantization methods and context sizes
1,024 tokens
Consumer
336x RTX 4090
24GB VRAM
Datacenter
76x NVIDIA A100
80GB VRAM
Apple Silicon
76x Apple M3 Max
128GB VRAM
10,000,000 tokens
Consumer
1288x RTX 4090
24GB VRAM
Datacenter
248x NVIDIA A100
80GB VRAM
Apple Silicon
316x Apple M3 Max
128GB VRAM
Architecture Diagram
Evaluation Benchmarks
No evaluation benchmarks for Llama 4 Behemoth available.
Rankings
Overall Rank
-
Coding Rank
-
About Llama 4 Behemoth
Llama 4 Behemoth is a large-scale multimodal foundation model developed by Meta, designed to serve as the primary teacher model within the Llama 4 family. As a non-deployed frontier model, its principal function is to generate high-quality synthetic data and provide the knowledge base for distilling smaller, production-ready variants such as Llama 4 Maverick and Scout. It integrates a native multimodal architecture capable of processing interleaved sequences of text, images, and video through an early fusion mechanism, which unifies visual and linguistic tokens within a single transformer backbone rather than utilizing separate modality-specific encoders.
The model utilizes a sparse Mixture-of-Experts (MoE) architecture to achieve a total parameter count of approximately 2 trillion. During inference, the routing mechanism activates a subset of approximately 288 billion parameters across 16 experts. Technical innovations include the use of Grouped-Query Attention (GQA) to manage memory bandwidth and a training regime optimized with FP8 precision on large-scale GPU clusters. The model's architecture incorporates interleaved attention layers and a novel distillation loss function designed to balance soft and hard targets during the knowledge transfer process to student models.
Developed as a research-centric artifact, Llama 4 Behemoth is optimized for complex reasoning tasks, mathematical problem-solving, and cross-modal understanding. By processing over 30 trillion tokens of diverse data, it establishes a high-capacity latent space that supports the training of highly efficient downstream models. While the model remains in a research preview status, its architectural design provides the technical foundation for the broader Llama 4 ecosystem, emphasizing scalability through sparsity and native cross-modal integration.
Technical Specifications
Attention
Attention Structure
Grouped-Query Attention
Attention Heads
128
Key-Value Heads
8
Attention Head Dimension
-
Position Embedding
Absolute Position Embedding
RoPE Theta
-
Sliding Window Attention
-
Sliding Window Size
-
Sliding Window Ratio
-
Linear Attention
-
Linear Attention Ratio
-
Normalization
RMS Normalization
Activation Function
SwigLU
Dimensions
Hidden Dimension Size
16,384
Number of Layers
160
FFN Intermediate Size (Dense)
-
Multi-Token Prediction Heads
-
Tokenizer
Vocabulary Size
-
Mixture of Experts
Total Expert Parameters
288.0B
Number of Experts
16
Active Experts
2
Shared Experts
-
FFN Intermediate Size (per Expert)
-
Dense Layers Before MoE
-
Resources
About Llama 4
Meta's Llama 4 model family implements a Mixture-of-Experts (MoE) architecture for efficient scaling. It features native multimodality through early fusion of text, images, and video. This iteration also supports significantly extended context lengths, with models capable of processing up to 10 million tokens.