ERNIE-4.5-21B-A3B

The model's architecture is extensively documented in the ERNIE 4.5 Technical Report. It is a Mixture-of-Experts (MoE) system with 21B total parameters and 3B active parameters. The report details a 'multimodal heterogeneous MoE' structure with 64 experts per layer, 6 active experts per token, and 2 shared experts. It specifies the use of Grouped-Query Attention (GQA), Rotary Position Embeddings (RoPE) with progressive frequency scaling, and a 28-layer depth. The training methodology, including the transition from multimodal pre-training to modality-specific post-training (SFT, DPO, and UPO), is clearly described.

While the technical report mentions a multi-stage training process involving 'trillions of tokens' and general categories like text and visual data, it lacks a specific quantitative breakdown of the dataset composition (e.g., percentages of web, code, or academic data). The documentation describes high-level cleaning and filtering strategies but does not disclose specific data sources or provide a detailed methodology for data collection, citing a 'human-model-in-the-loop' refinement cycle without granular transparency.

The tokenizer is publicly available via the 'tokenization_ernie4_5.py' file on Hugging Face and is based on SentencePiece. The vocabulary size and special tokens (e.g., <mask:1>, <s>, </s>) are explicitly defined in the code. It supports a 128K context window, and the implementation details for tokenization, including the use of alpha parameters for sampling and the handling of special tokens, are fully transparent and verifiable through the provided source code.

Baidu provides exemplary transparency regarding parameter density. The model is explicitly marketed as 21B total / 3B active, and the technical report further breaks this down into 64 total experts per layer with 6 activated, plus 2 shared experts. The architectural configuration (28 layers, 20 query heads, 4 KV heads) is precisely stated, leaving no ambiguity about the sparse vs. dense nature of the model.

The technical report mentions the use of NVIDIA H800 GPUs and the PaddlePaddle framework, achieving 47% Model FLOPs Utilization (MFU). However, it does not disclose the total GPU hours, the specific duration of the training run for the 21B variant, or the estimated carbon footprint. While it mentions the infrastructure's efficiency, the lack of concrete resource consumption metrics prevents a higher score.

The model provides results on standard benchmarks like BBH, CMATH, IFEval, and SimpleQA. While the technical report names these benchmarks and compares them against competitors like Qwen3, it does not provide the exact evaluation code or the specific prompts/few-shot examples used for all tests. Third-party testing is available on platforms like Hugging Face and YouTube, but official reproduction instructions are limited to general API usage snippets.

The model consistently identifies as part of the ERNIE 4.5 family across documentation, Hugging Face cards, and GitHub. It distinguishes clearly between its 'Base', 'PT' (Post-trained), and 'Thinking' variants. There is no evidence of identity confusion or claims of being a competitor's model in its official documentation or technical report.

The model is released under the Apache License 2.0, which is a standard, highly permissive open-source license. This is explicitly stated on the official Hugging Face repository, the GitHub repository, and the technical report. The license terms allow for commercial use and derivative works without conflicting proprietary restrictions.

Hardware requirements are well-documented. Official sources and third-party guides specify that approximately 48GB of VRAM is required for FP16 inference, recommending 2x RTX 4090 or 1x A6000. The documentation also highlights support for 4-bit and 2-bit 'convolutional code quantization' to reduce memory footprint, providing a clear path for deployment on various hardware tiers.

The model uses a naming convention that indicates its family and variant (e.g., ERNIE-4.5-21B-A3B-PT), but there is no evidence of a formal semantic versioning system or a detailed public changelog for weight updates. While new variants like the 'Thinking' model are announced via blogs, tracking minor updates or behavioral drift over time remains difficult for external users.