Qwen2.5-3B

The Qwen2.5-3B architecture is comprehensively documented in the official technical report and Hugging Face model cards. It is a dense, decoder-only Transformer utilizing Grouped-Query Attention (GQA) with 16 query heads and 2 KV heads, SwiGLU activation, RMSNorm, and Rotary Positional Embeddings (RoPE). The model specifies 36 layers and an embedding dimension of 2048. While the training methodology (pre-training followed by SFT and RLHF/GRPO) is described, the specific hyperparameters for the 3B variant's training run are less detailed than the flagship 72B model.

Alibaba discloses that the model was trained on 18 trillion tokens, a significant increase from previous versions. However, the exact composition is described only in general categories: high-quality web data, code, and mathematics. While they mention filtering and the use of synthetic data generated by larger Qwen models for math and code, they do not provide a precise percentage breakdown (e.g., web: X%, code: Y%) or name specific data sources, citing quality curation processes instead of providing a full provenance.

The tokenizer is publicly available via the 'qwen.tiktoken' and Hugging Face 'tokenization_qwen2.py' files. It uses Byte-Level Byte Pair Encoding (BBPE) with a large vocabulary of 151,643 regular tokens. Documentation explicitly states its efficiency for multilingual support (29+ languages) and provides compression rate comparisons. The vocabulary is consistent across the entire Qwen2.5 family, and the approach to handling control tokens is well-documented.

The parameter count is precisely disclosed as 3.09 billion total parameters, with 2.77 billion non-embedding parameters. As a dense model, all parameters are active during inference, which is clearly stated. The architectural breakdown (layers, heads, dimensions) is fully provided in the model configuration files and technical report, leaving no ambiguity regarding its density or structure.

Information regarding the specific compute resources used to train the 3B variant is largely absent. While the technical report mentions the use of large-scale GPU clusters for the series, it does not disclose the specific GPU hours, hardware type (e.g., H100 vs A100), or the carbon footprint associated with the 3B model's training. This is a significant gap compared to Western counterparts like Llama 3.1.

While Alibaba provides extensive benchmark results across standard sets (MMLU, HumanEval, MATH), they do not provide the exact evaluation code or the specific prompts/few-shot templates used for the 3B variant. Third-party researchers have raised significant concerns regarding data contamination in the Qwen2.5 series, particularly in mathematical benchmarks, which Alibaba has not addressed with a public audit or contamination analysis for this specific model.

The model consistently identifies itself as part of the Qwen series and is transparent about its versioning (2.5). It does not exhibit the identity confusion seen in some other models (e.g., claiming to be GPT-4). The model card and system prompts are designed to maintain a clear identity, and the model is generally aware of its capabilities and limitations as a 3B parameter model.

The model is released under the 'Qwen Research License Agreement'. While the terms are publicly accessible, it is not a standard Open Source license like Apache 2.0 (which is used for other sizes in the same family). The license includes restrictions on commercial use (requiring a separate request for a commercial license) and contains 'Materials' definitions that can be legally complex, creating more friction than standard permissive licenses.

VRAM requirements are well-documented by both the provider and the community. Official documentation notes support for context lengths up to 128K, with clear guidance on memory scaling. Quantization support (GPTQ, AWQ, GGUF) is extensively documented with performance/memory trade-offs provided in the technical report and community benchmarks, making it easy for users to estimate hardware needs.

Alibaba uses a versioning system (Qwen1.5, Qwen2, Qwen2.5), but detailed changelogs for minor updates or weight refreshes are often missing. There is no formal mechanism for tracking 'silent' updates to the weights on Hugging Face, and while the major versions are distinct, the lack of a granular versioning history for the 3B variant makes it difficult to track behavioral drift over time.