ERNIE-4.5-21B-A3B
Active Parameters
21B
Context Length
131.072K
Modality
Text
Architecture
Mixture of Experts (MoE)
License
Apache 2.0
Release Date
30 Jun 2025
Knowledge Cutoff
-
Technical Specifications
Total Expert Parameters
3.0B
Number of Experts
64
Active Experts
6
Attention Structure
Grouped-Query Attention
Hidden Dimension Size
-
Number of Layers
28
Attention Heads
20
Key-Value Heads
4
Activation Function
-
Normalization
-
Position Embedding
Absolute Position Embedding
System Requirements
VRAM requirements for different quantization methods and context sizes
ERNIE-4.5-21B-A3B
ERNIE 4.5 is a family of large-scale models developed by Baidu, designed to advance multimodal and language understanding. The ERNIE-4.5-21B-A3B variant is a Mixture-of-Experts (MoE) model with 21 billion total parameters and 3 billion activated parameters per token, specifically optimized for text understanding and generation tasks.
This model incorporates a heterogeneous MoE structure that allows for parameter sharing across modalities while also supporting dedicated parameters for individual modalities, which is intended to enhance multimodal understanding without compromising text-related task performance. The architecture employs a fine-grained MoE backbone, routing text inputs to distinct expert sets to mitigate cross-modal interference. A subset of shared experts and all self-attention layers are maintained for all tokens to facilitate cross-modal knowledge integration. Additionally, it features a modality-aware expert allocation strategy where visual experts are proportioned to optimize visual information processing. The training infrastructure is designed for scalability and efficiency, utilizing heterogeneous hybrid parallelism, hierarchical load balancing, FP8 mixed-precision training, and fine-grained recomputation methods for high pre-training throughput.
For inference, ERNIE 4.5 models incorporate multi-expert parallel collaboration and convolutional code quantization algorithms, supporting 4-bit/2-bit near-lossless quantization. The model supports an extended context length of up to 131,072 tokens. It is designed for efficient deployment across various hardware platforms and integrates with tools like ERNIEKit for fine-tuning methods such as Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Unified Preference Optimization (UPO). The models are accessible via the PaddlePaddle deep learning framework, which also supports PyTorch weights for the ERNIE-4.5-21B-A3B-PT variant.
About ERNIE 4.5
The Baidu ERNIE 4.5 family consists of ten large-scale multimodal models. They utilize a heterogeneous Mixture-of-Experts (MoE) architecture, which enables parameter sharing across modalities while also employing dedicated parameters for specific modalities, supporting efficient language and multimodal processing.
Other ERNIE 4.5 Models
Evaluation Benchmarks
Ranking is for Local LLMs.
No evaluation benchmarks for ERNIE-4.5-21B-A3B available.
Rankings
Overall Rank
-
Coding Rank
-
GPU Requirements
Full Calculator
Choose the quantization method for model weights
Context Size: 1,024 tokens