ERNIE-4.5-300B-A47B

The model's architecture is extensively documented in the ERNIE 4.5 Technical Report (2025). It details a novel 'multimodal heterogeneous MoE' structure with 54 Transformer layers and 64 total experts (8 active per token). The report specifies the use of Grouped Query Attention (GQA) with 64 query heads and 8 KV heads, and modality-isolated routing to prevent cross-modal interference. While it is a 'from scratch' training rather than a fine-tune of a known base, the architectural modifications and pre-training procedures are transparently described.

The technical report mentions a diverse corpus including web documents, books, academic papers, and conversational data, but lacks a specific percentage breakdown (e.g., exact ratios of code vs. web). It provides good detail on the 'human-model-in-the-loop' iterative refinement process and the use of over 10 specialized NLP models for quality scoring (toxicity, factuality), but the specific data sources remain generalized under 'public and in-house data' without naming specific datasets.

The tokenizer is publicly accessible via the official GitHub (ERNIEKit) and Hugging Face repositories. It uses a SentencePiece-based approach with a documented vocabulary size of 103,424 tokens. The implementation details, including special tokens (e.g., mask tokens) and BPE pre-tokenization support, are verifiable through the provided source code and configuration files.

Baidu is highly transparent regarding parameter density, explicitly stating the 300B total parameter count and the 47B active parameter count for this MoE variant. The technical report further breaks down the expert structure (64 total, 8 active) and the heterogeneous allocation (vision experts being one-third the size of text experts), which is exemplary for MoE transparency.

The technical report discloses the hardware used (2016 NVIDIA H800 GPUs) and achieves a 47% Model FLOPs Utilization (MFU). However, it fails to provide the total training duration in hours or a calculated carbon footprint, which are required for a high score. It mentions 'optimal efficiency' and 'limited compute resources' but lacks the raw temporal data for independent cost/impact verification.

Baidu provides extensive benchmark results across 28 datasets (MMLU-Pro, IFEval, GSM8K, etc.) and specifies versioning (e.g., DeepSeek-V3-0324 comparison). While evaluation code is partially available through the ERNIEKit repository, the exact prompts and few-shot examples for every benchmark are not fully disclosed in a single reproducible manifest, and third-party verification is still pending for the most recent 4.5 claims.

The model exhibits high identity consistency, correctly identifying as ERNIE 4.5 across various interfaces. It maintains clear versioning within the family (e.g., distinguishing between the 300B-A47B and the 21B-A3B variants). There are no documented cases of the model claiming to be a competitor's system or misrepresenting its MoE nature.

The model is released under a clear, standard Apache 2.0 license, which is explicitly stated in the technical report, GitHub repository, and Hugging Face model cards. This license permits both research and commercial use without the restrictive 'custom' terms often found in other large-scale model releases.

Hardware requirements are well-documented, with specific VRAM guidance for different precisions (e.g., 4x 80GB GPUs for 4-bit, 1x 141GB for 2-bit). The documentation also details the impact of quantization (W4A8C8, WINT2) and provides deployment scripts via FastDeploy. Scaling for the 131k context window is mentioned but lacks a detailed memory-per-token scaling chart.

Baidu uses a clear naming convention for variants, but a formal, public-facing changelog tracking specific weight updates or 'silent' alignment shifts is not prominently maintained. While major versions are distinct, the granular tracking of model drift over time is less transparent than the initial release documentation.