ERNIE-4.5-21B-A3B-Base

The model's architectural details are extensively documented in the ERNIE 4.5 Technical Report and official Hugging Face model cards. It is explicitly identified as a text-focused Mixture-of-Experts (MoE) variant derived from a larger multimodal pre-training phase. Key architectural components such as the 28-layer structure, Grouped-Query Attention (GQA) with 20 query and 4 KV heads, and the use of 2D Rotary Position Embeddings (RoPE) with progressive frequency scaling (10K to 500K) are clearly stated. The 'modality-specific extraction' process from the multimodal base is well-described, providing high clarity on its origin.

While the technical report mentions training on 'trillions of tokens' and 'diverse multimodal data sources,' it lacks a granular breakdown of the dataset composition (e.g., specific percentages of web, code, or academic data). It describes a 'REEAO' data processing system that ensures bitwise-deterministic token sequences and prevents duplication, which provides some methodological transparency. However, the specific sources and their proportions remain largely undisclosed, falling into the category of general data categories without specific verification of the underlying corpus.

The tokenizer is publicly accessible via the 'tokenization_ernie4_5.py' script on Hugging Face and is based on SentencePiece. The vocabulary size and special tokens (e.g., <mask:1>, <mask:7>) are explicitly defined in the code. Documentation confirms support for a 131,072-token context window, and the tokenizer is compatible with standard libraries like Hugging Face Transformers (v4.54.0+), allowing for direct public verification of its behavior and alignment with claimed language support.

Baidu provides precise figures for parameter density: 21 billion total parameters with approximately 3 billion active parameters per forward pass. The MoE structure is detailed as having 64 experts per layer, with 6 experts activated per token plus 2 shared experts. This level of detail—distinguishing between total, active, shared, and specialized experts—is exemplary for an MoE model and avoids the common pitfall of misleading total-parameter marketing.

The documentation mentions the use of Baidu's PaddlePaddle framework and hardware-level optimizations like FP8 mixed-precision training. While it references a large-scale cluster (30,000 Kunlun chips) and mentions achieving 47% Model FLOPs Utilization (MFU), it does not provide the specific total GPU/TPU hours or the exact hardware count used for this specific 21B variant's training run. Carbon footprint and specific cost estimates are also absent.

The technical report lists performance across various benchmarks (BBH, CMATH, GSM8K, HumanEval+) and compares them to competitors like Qwen3. While the evaluation results are detailed, the specific evaluation code and exact prompts used for every benchmark are not fully centralized in a single reproducible repository, though some integration with standard evaluation frameworks is implied. Third-party verification is starting to appear on leaderboards like SnakeBench, but a comprehensive reproduction guide is missing.

The model consistently identifies as part of the ERNIE 4.5 family across all official documentation, GitHub repositories, and API providers. It maintains clear versioning (e.g., the 'Thinking' vs 'Base' variants) and does not exhibit identity confusion with other models like GPT-4. The capabilities and limitations regarding its text-only nature (extracted from a multimodal base) are transparently communicated.

The model is explicitly released under the Apache License 2.0, which is a standard, highly permissive open-source license. This is clearly stated on Hugging Face, GitHub, and in the technical report. The license permits commercial use and derivative works without conflicting proprietary terms, providing a high degree of legal transparency for developers.

Hardware requirements are well-documented, with specific VRAM estimates provided for different precisions (e.g., ~48GB for FP16/BF16). The documentation explicitly mentions support for 4-bit and 2-bit 'lossless' quantization using convolutional code quantization, and provides guidance on the necessary GPU resources (e.g., 2x RTX 4090 or 1x A6000). Context length memory scaling is also addressed through the mention of FlashMask and optimized memory scheduling.

Baidu uses a clear naming convention (ERNIE-4.5-21B-A3B-Base) and maintains a changelog for its ERNIEKit and FastDeploy tools on GitHub. However, the frequency of model weight updates and a formal semantic versioning system for the weights themselves (to track potential silent drift) are less clearly defined compared to the software tools. There is a history of 'Thinking' upgrades, but a centralized, detailed weight-level changelog is not fully evident.