Active Parameters
300B
Context Length
131K
Modality
Text
Architecture
Mixture of Experts
KV Cache Rate
216 KB/tok
License
Apache 2.0
Release Date
30 Jun 2025
Knowledge Cutoff
Mar 2025
API Pricing (per 1M)
Input: $0.28 · Output: $1.10
Precision
Context
Workload Memory: 650.53 GB
FP16 · 8K context
Base Weights: 646.34 GB
KV Cache: 1.69 GB
Runtime & Graphs: 2.50 GB
8x NVIDIA H200 141GB
650.5 / 1015.2 GB usable · TP 8
Determine requirements for your specific workload:
Upgrade to Pro
Checkpoint format
FP16
CUDA
>= 11.8
PyTorch
>= 2.1
Transformers
>= 4.54.0.dev0
| Architecture | Precision | Attention kernel |
|---|---|---|
Blackwell, Hopper | Native FP16 | FlashAttention-3 |
Ada Lovelace, Ampere | Native FP16 | FlashAttention-2 |
Rank
#206
| Benchmark | Score | Rank |
|---|---|---|
Intelligence Index | 0.07 | 274 |
Overall Rank
#206
Coding Rank
-
ERNIE-4.5-300B is Baidu's large-scale text Mixture-of-Experts model activating 47B parameters for advanced English and Chinese natural language reasoning. It supports 4-bit and 2-bit quantization for enterprise-grade deployment over a 131K context.
Attention
Attention Structure
Grouped-Query Attention
Attention Heads
64
Key-Value Heads
8
Position Embedding
Absolute Position Embedding
RoPE Theta
500,000
Normalization
RMS Normalization
Activation Function
Swish
Dimensions
Base Precision
FP16
Hidden Dimension Size
8,192
Number of Layers
54
FFN Intermediate Size (Dense)
3,584
Multi-Token Prediction Heads
1
Tokenizer
Vocabulary Size
103,424
Mixture of Experts
Total Expert Parameters
47.0B
Number of Experts
64
Active Experts
8
Shared Experts
0
FFN Intermediate Size (per Expert)
3,584
Dense Layers Before MoE
3
The Baidu ERNIE 4.5 family consists of ten large-scale multimodal models. They utilize a heterogeneous Mixture-of-Experts (MoE) architecture, which enables parameter sharing across modalities while also employing dedicated parameters for specific modalities, supporting efficient language and multimodal processing.