Hunyuan Standard

The model architecture is extensively documented in a technical report (arXiv:2411.02265). It specifies a Transformer-based Mixture-of-Experts (MoE) design with 389B total and 52B active parameters. Key architectural details are disclosed, including the use of Rotary Position Embedding (RoPE), SwiGLU activation, Grouped-Query Attention (GQA), and a novel Cross-Layer Attention (CLA) mechanism. The report also details the expert configuration (1 shared expert and 16 specialized experts, with 1 activated per token) and the routing strategy. While the training methodology is described, the specific 'from scratch' vs. 'continued pre-training' lineage of the base architecture (beyond being a standard Transformer) is clear but lacks some historical development context.

Tencent discloses that the model was pre-trained on 7 trillion tokens, including a significant portion (1.5 trillion tokens) of synthetic data. The report mentions general categories such as web text, financial reports, legal documents, and academic papers. However, it lacks a precise percentage breakdown of the dataset composition (e.g., exact ratios of code vs. web vs. books) and does not provide specific names or links to the 5.5 trillion tokens of natural data sources, citing them as 'unspecified sources' in third-party analysis. The data cleaning and filtering pipeline is mentioned but not documented with reproducible detail.

The tokenizer is publicly available via the official GitHub and Hugging Face repositories. It uses a 128K vocabulary size, combining 100K tokens from the tiktoken (OpenAI) base with an additional 28K-29K tokens specifically trained on high-quality Chinese data to improve compression. The documentation provides specific compression rate comparisons (3.13 characters per token vs. Llama 3.1's 2.78), and the tokenizer is fully inspectable through the released code.

The model provides exemplary transparency regarding its parameter density. It clearly distinguishes between the 389 billion total parameters and the 52 billion active parameters used during inference. The MoE structure is well-defined, specifying the number of experts (16 specialized + 1 shared) and the activation frequency. This prevents the common 'parameter inflation' marketing trap seen in other MoE models.

Information regarding training compute is extremely limited. While Tencent mentions using their 'Xingmai' high-performance network and NVIDIA H20 GPUs (due to export restrictions), they do not disclose the total GPU/TPU hours, the specific training duration, the total energy consumption, or the carbon footprint. Most compute-related information comes from general corporate announcements about their infrastructure rather than model-specific technical disclosures.

The technical report provides results for dozens of standard benchmarks (MMLU, GSM8K, MATH, etc.) and specifies the versions used (e.g., MMLU-Pro). Evaluation code for some specialized benchmarks like AutoCodeBench is public. However, the exact prompts and few-shot configurations for all standard benchmarks are not fully detailed in a single reproducible repository, and third-party verification is currently limited to leaderboard positions rather than independent reproduction of the full training-to-eval pipeline.

The model consistently identifies as 'Hunyuan-Large' or 'Hunyuan-MoE-A52B' across all official documentation, weights, and code repositories. There is no evidence of identity confusion or claims of being a competitor's model. Versioning between 'Pretrain' and 'Instruct' variants is clearly maintained.

The model is released under the 'Tencent Hunyuan Community License Agreement.' While it allows for free use and distribution for many, it contains significant restrictions: it is not valid in the European Union, and it requires a separate license request for entities with over 100 million monthly active users. These geographic and scale-based restrictions mean it does not meet the criteria for a standard open-source license (like Apache 2.0), creating some legal complexity for global users.

Hardware requirements are documented in the GitHub repository and technical report. It provides VRAM guidance for fine-tuning (e.g., 32 GPUs for full fine-tuning, 8 for LoRA) and highlights the memory-saving benefits of CLA and FP8 quantization (50% reduction). While it lacks a comprehensive VRAM-to-context-length scaling table for all quantization levels (Q4, Q8, etc.), the provided guidance for H20/H100 clusters is specific and verifiable.

The model uses basic versioning (e.g., Hunyuan-A52B-Instruct-FP8), and updates are posted on Hugging Face and GitHub. However, there is no formal semantic versioning system or a detailed public changelog documenting specific weight updates or behavioral drift over time. The project is relatively new (late 2024), so long-term tracking of silent degradation is not yet possible, but the current infrastructure for version history is minimal.