ERNIE-4.5-300B-A47B

Active Parameters

300B

Context Length

131K

Modality

Text

Architecture

Mixture of Experts

KV Cache Rate

216 KB/tok

License

Apache 2.0

Release Date

30 Jun 2025

Knowledge Cutoff

Mar 2025

API Pricing (per 1M)

Input: $0.28 · Output: $1.10

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 650.53 GB

FP16 · 8K context

Base Weights: 646.34 GB

KV Cache: 1.69 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

8x NVIDIA H200 141GB

650.5 / 1015.2 GB usable · TP 8

Determine requirements for your specific workload:

Upgrade to Pro

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.54.0.dev0

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: AbsoluteHidden: 8.2k · Context: 131K · Vocab: 103.4kx 54 layersRMSNormPre-AttentionGrouped-Query Attention64Q / 8KV headsHead dim: 128+RMSNormPre-FFNSparse MoE FFN (8/64 experts)SwishIntermediate: 3.6k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#206

BenchmarkScoreRank

Intelligence Index

Artificial Analysis

0.07

274

Rankings

Overall Rank

#206

Coding Rank

-

About ERNIE-4.5-300B-A47B

ERNIE-4.5-300B is Baidu's large-scale text Mixture-of-Experts model activating 47B parameters for advanced English and Chinese natural language reasoning. It supports 4-bit and 2-bit quantization for enterprise-grade deployment over a 131K context.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

64

Key-Value Heads

8

Position Embedding

Absolute Position Embedding

RoPE Theta

500,000

Normalization

RMS Normalization

Activation Function

Swish

Dimensions

Base Precision

FP16

Hidden Dimension Size

8,192

Number of Layers

54

FFN Intermediate Size (Dense)

3,584

Multi-Token Prediction Heads

1

Tokenizer

Vocabulary Size

103,424

Mixture of Experts

Total Expert Parameters

47.0B

Number of Experts

64

Active Experts

8

Shared Experts

0

FFN Intermediate Size (per Expert)

3,584

Dense Layers Before MoE

3

About ERNIE 4.5

The Baidu ERNIE 4.5 family consists of ten large-scale multimodal models. They utilize a heterogeneous Mixture-of-Experts (MoE) architecture, which enables parameter sharing across modalities while also employing dedicated parameters for specific modalities, supporting efficient language and multimodal processing.


Other ERNIE 4.5 Models