Qwen2.5-72B

The model architecture is explicitly documented as a dense decoder-only Transformer. Key components such as Grouped Query Attention (GQA), SwiGLU activation, RMSNorm, and Rotary Position Embeddings (RoPE) are detailed in the official technical report and blog posts. While the pre-training methodology is described as a staged process (initially 4k context, then 32k), the specific transition points and full hyperparameter sets for each stage are only partially disclosed. The model is a successor to Qwen2, and the evolution of architectural choices is well-documented.

Alibaba discloses that the model was trained on 18 trillion tokens, a significant increase from its predecessor. General categories are provided (web, books, code, math, and multilingual data), and the use of synthetic data generated by previous Qwen models is acknowledged. However, there is no specific percentage breakdown of the dataset composition (e.g., web vs. code), and the exact sources of the 'high-quality' data remain proprietary. Filtering and cleaning methodologies are mentioned but lack the granular detail required for a high score.

The tokenizer is publicly available via the Hugging Face repository and is well-documented. It uses Byte Pair Encoding (BPE) with a large vocabulary of 151,646 tokens, which is specifically designed to support over 29 languages without 'unknown' tokens. The vocabulary size and tokenization approach are consistent across official documentation and third-party implementations like vLLM and Ollama.

The model's parameter count is clearly stated as 72.7 billion total, with 70.0 billion non-embedding parameters. As a dense model, all parameters are active during inference, which is explicitly confirmed in the technical report. Detailed architectural specifications, including the number of layers (80), hidden dimension (12,288), and attention heads (64 Q, 8 KV), are publicly available.

Information regarding training compute is extremely limited. While the technical report mentions the use of 'large-scale distributed infrastructure' and provides some optimizer settings (AdamW, learning rate schedules), it fails to disclose the total GPU/TPU hours, the specific hardware cluster size, or the training duration. No carbon footprint or environmental impact data is provided, which is a significant transparency gap for a model of this scale.

Alibaba provides comprehensive benchmark results across standard sets (MMLU, GSM8K, HumanEval, etc.) and specifies the versions used (e.g., MMLU-Pro, LiveCodeBench). However, while the evaluation results are detailed, the exact evaluation code and full prompt templates used to generate these specific scores are not fully centralized or provided in a 'one-click' reproducible format. Third-party verification is available through leaderboards like LMSYS Chatbot Arena, which adds credibility but does not replace the need for official reproduction scripts.

The model consistently identifies itself as Qwen, developed by Alibaba Cloud, in its system prompts and official documentation. It maintains a clear versioning identity (Qwen2.5) and is transparent about its nature as an AI. There are no documented cases of the model claiming to be a competitor's product in its official weights.

The model is released under the 'Qwen License Agreement,' which is a custom license rather than a standard open-source license like Apache 2.0. While the terms are publicly accessible and clearly state that commercial use is permitted for entities with fewer than 100 million monthly active users, the requirement for a separate agreement beyond that threshold introduces a proprietary restriction. This 'open-weights but not open-source' distinction is clearly communicated but limits the score compared to truly permissive licenses.

Official documentation and the model card provide clear guidance on context length (128k tokens) and output limits. While Alibaba's own documentation on specific VRAM requirements for various quantization levels is somewhat sparse, the community and third-party providers (e.g., Hugging Face, Ollama) have extensively documented the VRAM needs for FP16 (~144GB), 8-bit (~77GB), and 4-bit (~47GB) versions. The model's memory scaling with context length is also well-understood by the community.

The model follows a clear naming convention (Qwen2 -> Qwen2.5), but detailed changelogs for minor weight updates or 'silent' refinements are not consistently maintained in a public-facing ledger. While the transition from Qwen2 to 2.5 was a major, well-documented event, there is limited transparency regarding the ongoing maintenance or potential behavior drift of the weights within the 2.5-72B-Instruct repository since its initial release.