Hunyuan Turbo
Active Parameters
52B (Estimated)
Context Length
32K
Modality
Text
Architecture
Mixture of Experts (MoE)
License
Proprietary
Release Date
15 May 2024
Knowledge Cutoff
Dec 2023
API Pricing (per 1M)
Input: $0.28 · Output: $0.28
Evaluation Benchmarks
Rank
#111
| Benchmark | Score | Rank |
|---|---|---|
General Text | 1341 | 110 |
Rankings
Overall Rank
#111
Coding Rank
-
About Hunyuan Turbo
Hunyuan Turbo is a high-concurrency enterprise Mixture-of-Experts model from Tencent combining Mamba state-space blocks with transformer attention. It delivers low-latency processing and dual execution paths for high-volume customer workflows.
Technical Specifications
Architecture specifications are undisclosed for proprietary models.
Attention
Attention Structure
Multi-Head Attention
Attention Heads
Undisclosed
Key-Value Heads
Undisclosed
Attention Head Dimension
Undisclosed
Position Embedding
Absolute Position Embedding
RoPE Theta
Undisclosed
Sliding Window Attention
Undisclosed
Sliding Window Size
Undisclosed
Sliding Window Ratio
Undisclosed
Linear Attention
Undisclosed
Linear Attention Ratio
Undisclosed
Normalization
RMS Normalization
Activation Function
SwigLU
Dimensions
Hidden Dimension Size
4,096
Number of Layers
Undisclosed
FFN Intermediate Size (Dense)
Undisclosed
Multi-Token Prediction Heads
Undisclosed
Tokenizer
Vocabulary Size
Undisclosed
Mixture of Experts
Total Expert Parameters
52.0B
Number of Experts
16
Active Experts
2
Shared Experts
Undisclosed
FFN Intermediate Size (per Expert)
Undisclosed
Dense Layers Before MoE
Undisclosed
Resources
About Hunyuan
Tencent Hunyuan large language models with various capabilities.