Hunyuan Standard
Active Parameters
52B
Context Length
30K
Modality
Text
Architecture
Mixture of Experts (MoE)
License
Tencent Hunyuan Community License Agreement
Release Date
10 Jun 2024
Knowledge Cutoff
-
API Pricing (per 1M)
Self-hosted only
System Requirements
VRAM requirements for different quantization methods and context sizes
1,024 tokens
Consumer
6x RTX 4090
24GB VRAM
Datacenter
2x NVIDIA A100
80GB VRAM
Apple Silicon
1x Apple M3 Max
128GB VRAM
30,000 tokens
Consumer
6x RTX 4090
24GB VRAM
Datacenter
2x NVIDIA A100
80GB VRAM
Apple Silicon
1x Apple M3 Max
128GB VRAM
Architecture Diagram
Evaluation Benchmarks
Rank
#143
| Benchmark | Score | Rank |
|---|---|---|
General Text | 1311 | 134 |
Rankings
Overall Rank
#143
Coding Rank
-
About Hunyuan Standard
Hunyuan-Large (MoE-A52B) is Tencent's open-source 389B Mixture-of-Experts model activating 52B parameters for advanced bilingual reasoning and text generation. It incorporates Cross-Layer Attention and Grouped-Query Attention across a 256K context window.
Technical Specifications
Attention
Attention Structure
Multi-Head Attention
Attention Heads
80
Key-Value Heads
8
Attention Head Dimension
-
Position Embedding
Absolute Position Embedding
RoPE Theta
-
Sliding Window Attention
-
Sliding Window Size
-
Sliding Window Ratio
-
Linear Attention
-
Linear Attention Ratio
-
Normalization
-
Activation Function
SwigLU
Dimensions
Hidden Dimension Size
6,400
Number of Layers
64
FFN Intermediate Size (Dense)
-
Multi-Token Prediction Heads
-
Tokenizer
Vocabulary Size
-
Mixture of Experts
Total Expert Parameters
389.0B
Number of Experts
17
Active Experts
2
Shared Experts
-
FFN Intermediate Size (per Expert)
-
Dense Layers Before MoE
-
About Hunyuan
Tencent Hunyuan large language models with various capabilities.