ERNIE-4.5-VL-424B-A47B

The model's architecture is extensively documented in the ERNIE 4.5 Technical Report (June 2025). It details a novel 'multimodal heterogeneous MoE' structure that differentiates between text and vision processing while sharing self-attention layers. Specific architectural components are disclosed, including a 630M parameter ViT encoder, 128 total experts (64 text, 64 vision), and the use of Grouped Query Attention (GQA) with 64 heads. The report also specifies the use of 2D RoPE for vision and absolute position embeddings for the backbone.

While the technical report mentions the model was trained on 'trillions of tokens' and 'high-quality Chinese textual and visual data,' it lacks a specific breakdown of dataset sources, proportions, or detailed filtering methodologies. The description remains at a high level (e.g., 'extensive pre-training on visual concepts'), which falls into the category of 'general data categories mentioned' without verifiable composition details.

The tokenizer is publicly accessible via the PaddlePaddle and Hugging Face repositories. Documentation specifies a vocabulary size of 103,424 tokens. The tokenizer is aligned with the model's multilingual (Chinese/English) focus and is integrated into standard inference frameworks like vLLM and FastDeploy, allowing for direct verification of tokenization behavior.

Baidu provides precise figures for both total and active parameters: 424 billion total parameters with 47 billion active parameters per token. The documentation further breaks down the MoE structure (128 experts, 8 active per modality) and the vision encoder (630M parameters). This level of detail for a sparse architecture is exemplary.

The technical report discloses the use of 2,016 NVIDIA H800 GPUs for pre-training the largest language model variant and mentions a Model FLOPs Utilization (MFU) of 47%. However, it does not provide the total training duration in hours, the specific energy consumption, or a calculated carbon footprint for the VL-424B variant specifically, leaving significant gaps in environmental transparency.

Baidu reports results on standard benchmarks (MathVista, MMMU, MMLU-Pro, GSM8K) and provides some evaluation details in the technical report. While they open-source the 'ERNIEKit' for fine-tuning and 'FastDeploy' for inference, the exact evaluation scripts and full prompt sets used to achieve the reported state-of-the-art scores are not fully centralized for one-click reproduction.

The model demonstrates high identity consistency, with clear versioning (ERNIE 4.5) and variant labeling (VL-424B-A47B). It distinguishes between 'thinking' and 'non-thinking' modes in its system prompts and API parameters. Documentation and model cards consistently reflect its capabilities as a multimodal MoE model without claiming identities of other providers.

The model weights and associated development toolkits (ERNIEKit, FastDeploy) are explicitly released under the Apache License 2.0, as verified across GitHub, Hugging Face, and the official technical report. This is a highly permissive, standard open-source license that clearly allows for both research and commercial use.

Hardware requirements are well-documented for various precisions. Documentation specifies that 8x 80GB GPUs are required for 4-bit quantization, while native BF16 requires significantly more (up to 16x 80GB or 8x 140GB). The impact of 2-bit and 4-bit 'lossless' quantization is discussed, and VRAM estimates are provided for different deployment scenarios.

Baidu uses a clear naming convention and versioning for the ERNIE 4.5 family. However, there is limited public information regarding a formal changelog for weight updates or a public policy for managing model drift over time. While the release is recent, the infrastructure for tracking long-term version history and deprecation is not as transparent as its architectural documentation.