Qwen2-72B

The Qwen2-72B model is well-documented as a dense, decoder-only Transformer. The technical report and official blog posts explicitly detail the use of Grouped Query Attention (GQA), SwiGLU activation, and Rotary Positional Embeddings (RoPE) with a modified base frequency for long-context support. While the architecture is clearly defined as a successor to Qwen1.5, the specific pre-training methodology (e.g., exact curriculum or stage-by-stage hyperparameter shifts) is described in general terms rather than exhaustive detail.

Alibaba discloses that the model was trained on 7 trillion tokens of multilingual data across 27+ languages. However, the breakdown of this data (e.g., specific percentages of web, code, books, or academic papers) is not publicly quantified. The documentation mentions 'high-quality' and 'meticulously curated' sources including web crawls and code (referencing CodeQwen1.5), but lacks a verifiable, detailed composition report or access to sample data for independent verification.

The tokenizer is publicly available via the official GitHub repository and Hugging Face. It uses byte-level Byte Pair Encoding (BPE) with a large vocabulary size of 151,646 tokens, which is documented to improve compression across multiple languages. The vocabulary size and tokenization approach are consistently reported across all official sources and are verifiable through the provided code and model files.

The model is explicitly stated to be a dense architecture with 72.7 billion total parameters (70.0 billion non-embedding). The architectural configuration is fully disclosed, including 80 layers, 64 query heads, and 8 KV heads for GQA. This level of detail is high, though it lacks a deeper breakdown of parameter distribution across specific components like attention vs. FFN beyond what can be inferred from the layer specs.

There is virtually no public disclosure regarding the specific compute resources used for Qwen2-72B. The technical report mentions that models were trained on 'powerful GPUs' but fails to provide GPU/TPU hours, hardware counts, training duration, or the associated carbon footprint. This information is conspicuously absent, likely for competitive reasons, resulting in a low score.

Alibaba provides extensive benchmark results across 16+ datasets (MMLU, GSM8K, HumanEval, etc.) and specifies the few-shot settings used (e.g., 5-shot for MMLU). However, the exact evaluation code and the specific prompts/few-shot examples used to achieve these scores are not fully public in a single reproducible repository. Third-party evaluations (like Open LLM Leaderboard) exist but sometimes show variance from official claims.

The model consistently identifies itself as Qwen and is transparent about its versioning (Qwen2 vs Qwen1.5). It does not exhibit the identity confusion seen in some other models (e.g., claiming to be GPT-4). It is clear about its nature as an AI developed by Alibaba and provides consistent version tracking across its weights and documentation.

The model uses the 'Tongyi Qianwen LICENSE AGREEMENT'. While the terms are publicly accessible and allow for commercial use, there is a significant restriction: entities with more than 100 million monthly active users must request a separate license. This 'open-weights' but not 'open-source' (per OSI standards) nature creates some ambiguity for large-scale commercial adoption compared to Apache 2.0 models.

VRAM requirements are well-documented by both the provider and the community. Official documentation notes the reduced KV cache size due to GQA, and community resources (like Hugging Face and vLLM) provide clear guidance on the ~144GB VRAM needed for BF16 and the benefits of 4-bit/8-bit quantization. However, official documentation on the specific accuracy-tradeoffs for various quantization levels is less comprehensive than community-driven data.

Alibaba uses a clear naming convention (Qwen1.5 -> Qwen2 -> Qwen2.5) and maintains a GitHub repository with some changelog information. However, there are reports of performance drift and 'alignment tax' in instruction-tuned versions that are not formally documented in a centralized changelog. Silent updates to weights are less common here than in API-only models, but the transition between minor versions lacks granular diffs.