ApX logoApX logo

Hunyuan 4 Preview

Active Parameters

770B

Context Length

1.05M

Modality

Text

Architecture

Mixture of Experts

KV Cache Rate

156 KB/tok

License

Apache 2.0

Release Date

27 Aug 2026

Knowledge Cutoff

-

API Pricing (per 1M)

Input: $0.83 · Output: $2.50

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 1529.23 GB

FP16 · 8K context

Base Weights: 1525.51 GB

KV Cache: 1.22 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

16x NVIDIA H200 141GB

1529.2 / 2030.4 GB usable · TP 8 · PP 2

Determine requirements for your specific workload:

Upgrade to Pro

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 5.16.2

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashMLA

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 6.1k · Context: 1.05M · Vocab: 120.8kx 78 layersRMSNormPre-AttentionDeepSeek Sparse Attention64Q / 8KV headsHead dim: 64+RMSNormPre-FFNSparse MoE FFN (8/256 experts)SwiGLUIntermediate: 2k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#35

BenchmarkScoreRank

Graduate-Level QA

GPQA

0.923

14

Web Development

WebDev Arena

1632

22

Agent Arena

Agent Arena

0.42

35

Intelligence Index

Artificial Analysis

0.35

61

Rankings

Overall Rank

#35

Coding Rank

#49

About Hunyuan 4 Preview

Hunyuan 4 Preview is Tencent's next-generation foundation model preview showcasing advanced Chinese and English language understanding. It features enhanced architectural scalability for enterprise reasoning and text generation tasks.

Specifications

Attention

Attention Structure

DeepSeek Sparse Attention

Attention Heads

64

Key-Value Heads

8

Attention Head Dimension

64

Position Embedding

ROPE

RoPE Theta

10,000,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

6,144

Number of Layers

78

FFN Intermediate Size (Dense)

18,432

Multi-Token Prediction Heads

1

Tokenizer

Vocabulary Size

120,832

Mixture of Experts

Total Expert Parameters

49.0B

Number of Experts

256

Active Experts

8

Shared Experts

1

FFN Intermediate Size (per Expert)

2,048

Dense Layers Before MoE

1