Qwen2.5-1.5B

The model's architecture is extensively documented in the Qwen2.5 Technical Report and official Hugging Face model cards. It is a dense, decoder-only Transformer utilizing Rotary Position Embeddings (RoPE), SwiGLU activation, RMSNorm, and Grouped Query Attention (GQA). Specific configurations for the 1.5B variant are provided, including 28 layers, a hidden dimension of 1536, and a GQA setup with 12 query heads and 2 key-value heads. The transition from the Qwen2 base is clearly stated, though the exact pre-training recipe (e.g., specific learning rate schedules or optimizer hyperparameters for this specific variant) is less detailed than the general series documentation.

While Alibaba discloses that the model was trained on a massive 18 trillion token dataset (an increase from 7 trillion in Qwen2), the specific composition breakdown is vague. Documentation mentions general categories like 'web data', 'code', and 'mathematics' and notes the inclusion of 29+ languages. However, it lacks a precise percentage breakdown (e.g., web: X%, code: Y%) or a list of specific data sources. The use of synthetic data is acknowledged, particularly for the Coder and Math variants, but the exact ratio for the base 1.5B model remains undisclosed.

The tokenizer is publicly accessible via the 'qwen.tiktoken' file and Hugging Face's 'tokenization_qwen2.py'. It uses Byte Pair Encoding (BPE) with a large vocabulary of 151,646 tokens, which is well-documented for its efficiency across multiple languages (English, Chinese, etc.). The vocabulary size and special tokens (like <|endoftext|>) are clearly defined, and the alignment with the model's multilingual claims is verifiable through the public code and configuration files.

The parameter count is precisely stated as 1.54 billion total and 1.31 billion non-embedding parameters. As a dense model, all parameters are active during inference, avoiding the 'active vs total' ambiguity found in MoE models. The architectural breakdown (layers, heads, hidden dims) is fully transparent in the technical report and model config files, allowing for a clear understanding of parameter distribution.

Information regarding the training compute is extremely limited. While the technical report mentions the scale of the data (18T tokens), it does not disclose the specific hardware used (e.g., number of H100/A100 GPUs), the total GPU hours consumed, or the estimated carbon footprint. Some third-party research has attempted to estimate the energy footprint for inference, but official training compute metrics are conspicuously absent for 'competitive reasons'.

Alibaba provides comprehensive benchmark results across standard sets like MMLU, HumanEval, and MATH in their technical report. However, the score is moderated because the exact evaluation prompts, few-shot examples, and specific code used to generate these scores are not fully public in a single reproducible repository. While some evaluation scripts are available in the QwenLM GitHub, they do not cover the full breadth of the reported results, making exact third-party reproduction difficult.

The model demonstrates high identity consistency, correctly identifying itself as part of the Qwen series in most standard deployments. It maintains clear versioning (2.5) and distinguishes between its base and 'Instruct' variants. There are no significant reports of the model claiming to be a competitor's product (e.g., GPT-4) or denying its nature as an AI developed by Alibaba.

The Qwen2.5-1.5B model is explicitly released under the Apache 2.0 license, which is a highly permissive, standard open-source license. This is clearly stated on the Hugging Face repository and the official blog. This marks a transparent shift from earlier versions that used the more restrictive 'Tongyi Qianwen License', providing clear terms for both commercial and research use.

Hardware requirements are well-documented by both the official team and the community. Official documentation provides VRAM estimates for different context lengths and quantization levels (FP16, INT8, INT4). For example, the 1.5B model is noted to require ~3-4GB VRAM for FP16 inference. Third-party tools like vLLM and Ollama further validate these requirements, though official documentation on the specific accuracy-performance trade-offs of quantization is less detailed.

Alibaba uses a clear versioning system (Qwen -> Qwen1.5 -> Qwen2 -> Qwen2.5). However, the score is limited because detailed changelogs for minor weight updates or 'silent' refinements are not always provided. While major releases are well-documented, the community has noted occasional issues with EOS tokens and chat templates in base models that required manual fixes, indicating some gaps in the official versioning and release verification process.