技术规格

系统要求

不同量化方法和上下文大小的显存要求

ERNIE-4.5-21B-A3B

ERNIE 4.5 is a family of large-scale models developed by Baidu, designed to advance multimodal and language understanding. The ERNIE-4.5-21B-A3B variant is a Mixture-of-Experts (MoE) model with 21 billion total parameters and 3 billion activated parameters per token, specifically optimized for text understanding and generation tasks.

This model incorporates a heterogeneous MoE structure that allows for parameter sharing across modalities while also supporting dedicated parameters for individual modalities, which is intended to enhance multimodal understanding without compromising text-related task performance. The architecture employs a fine-grained MoE backbone, routing text inputs to distinct expert sets to mitigate cross-modal interference. A subset of shared experts and all self-attention layers are maintained for all tokens to facilitate cross-modal knowledge integration. Additionally, it features a modality-aware expert allocation strategy where visual experts are proportioned to optimize visual information processing. The training infrastructure is designed for scalability and efficiency, utilizing heterogeneous hybrid parallelism, hierarchical load balancing, FP8 mixed-precision training, and fine-grained recomputation methods for high pre-training throughput.

For inference, ERNIE 4.5 models incorporate multi-expert parallel collaboration and convolutional code quantization algorithms, supporting 4-bit/2-bit near-lossless quantization. The model supports an extended context length of up to 131,072 tokens. It is designed for efficient deployment across various hardware platforms and integrates with tools like ERNIEKit for fine-tuning methods such as Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Unified Preference Optimization (UPO). The models are accessible via the PaddlePaddle deep learning framework, which also supports PyTorch weights for the ERNIE-4.5-21B-A3B-PT variant.

没有可用的 ERNIE-4.5-21B-A3B 评估基准。