ERNIE-4.5-300B-A47B-Base

System Requirements

VRAM requirements for different quantization methods and context sizes

No evaluation benchmarks for ERNIE-4.5-300B-A47B-Base available.

About ERNIE-4.5-300B-A47B-Base

The ERNIE-4.5-300B-A47B-Base model, developed by Baidu, is a large-scale language model utilizing a Mixture-of-Experts (MoE) architecture. As a prominent member of the ERNIE 4.5 family, it contains 300 billion total parameters while activating 47 billion parameters per token through a sparse gated mechanism. This design allows the model to scale its knowledge capacity significantly without a linear increase in per-token inference costs. The model is specifically optimized for advanced text-based reasoning, code generation, and complex instruction following across both English and Chinese languages.

Technically, the model introduces a multimodal heterogeneous MoE structure, which was pre-trained on trillions of tokens using a joint textual and visual modality framework. A key architectural innovation is the modality-isolated routing technique, which ensures that expert specialization for one modality does not negatively impact the performance of another. This variant, the A47B-Base, represents the extracted text-related parameters following this extensive multimodal pre-training phase. It employs Grouped Query Attention (GQA) with 64 query heads and 8 key-value heads to achieve a balance between attention quality and memory efficiency during long-context processing.

The architecture is built upon the PaddlePaddle deep learning framework and supports an expansive context window of 131,072 tokens. To manage the computational demands of a 300B parameter system, Baidu implemented scaling-efficient infrastructure features such as intra-node expert parallelism, memory-efficient pipeline scheduling, and FP8 mixed-precision training. The model is designed for high-throughput deployment environments and supports advanced inference optimizations, including Prefill-Decode (PD) disaggregation with dynamic role switching to maximize hardware utilization.

Technical Specifications

Resources

About ERNIE 4.5

The Baidu ERNIE 4.5 family consists of ten large-scale multimodal models. They utilize a heterogeneous Mixture-of-Experts (MoE) architecture, which enables parameter sharing across modalities while also employing dedicated parameters for specific modalities, supporting efficient language and multimodal processing.

Other ERNIE 4.5 Models

• ERNIE-4.5-0.3B
• ERNIE-4.5-0.3B-Base
• ERNIE-4.5-21B-A3B
• ERNIE-4.5-21B-A3B-Base
• ERNIE-4.5-VL-28B-A3B
• ERNIE-4.5-VL-28B-A3B-Base
• ERNIE-4.5-300B-A47B
• ERNIE-4.5-VL-424B-A47B
• ERNIE-4.5-VL-424B-A47B-Base