Hunyuan A13B

Active Parameters

80B

Context Length

256K

Modality

Text

Architecture

Mixture of Experts

KV Cache Rate

128 KB/tok

License

Apache 2.0

Release Date

25 Jun 2025

Knowledge Cutoff

-

API Pricing (per 1M)

Self-hosted only

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 176.73 GB

FP16 · 8K context

Base Weights: 173.23 GB

KV Cache: 1.00 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

8x NVIDIA H100 80GB

176.7 / 576.0 GB usable · TP 8

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.41.2

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Additional required packages

Install alongside standard runtime libraries:

pip install flash_attn

Architecture Diagram

Input TokensToken EmbeddingPosition: AbsoluteHidden: 4.1k · Context: 256K · Vocab: 128.2kx 32 layersRMSNormPre-AttentionMulti-Head Attention32Q / 8KV headsHead dim: 128+RMSNormPre-FFNSparse MoE FFN (8/65 experts)SwiGLUIntermediate: 3.1k+Final RMSNormOutput Logits

Evaluation Benchmarks

No evaluation benchmarks for Hunyuan A13B available.

Rankings

Overall Rank

-

Coding Rank

-

About Hunyuan A13B

Hunyuan A13B is an open-source 80B Mixture-of-Experts model by Tencent activating 13B parameters for efficient STEM reasoning and agentic tasks. It features dual fast/slow thinking modes and supports quantization down to INT4 across a 256K context.

Specifications

Attention

Attention Structure

Multi-Head Attention

Attention Heads

32

Key-Value Heads

8

Attention Head Dimension

128

Position Embedding

Absolute Position Embedding

RoPE Theta

10,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

4,096

Number of Layers

32

FFN Intermediate Size (Dense)

3,072

Tokenizer

Vocabulary Size

128,167

Mixture of Experts

Total Expert Parameters

13.0B

Number of Experts

65

Active Experts

8

Shared Experts

1

FFN Intermediate Size (per Expert)

3,072

Hunyuan A13B - VRAM Requirements, Specs & Benchmarks