Qwen2-1.5B

Qwen2-1.5B is explicitly documented as a dense, decoder-only Transformer model. The technical report and official blog posts detail the use of SwiGLU activation, RoPE (Rotary Positional Embeddings), RMSNorm, and Group Query Attention (GQA). It is a from-scratch pre-trained model (not a fine-tune of a competitor base), and the transition from Qwen1.5 is documented. While the high-level architecture is clear, specific layer-by-layer configuration details are primarily found in the code/config files rather than a centralized architectural paper.

The training data is described as a 'high-quality, large-scale dataset' of 7 trillion tokens (for the 1.5B variant). While the technical report mentions broad categories like web data, code, and mathematics, and notes an increase in multilingual data (29+ languages), there is no specific percentage breakdown of the mixture (e.g., % web vs % code). The data collection and filtering methodologies are described in vague terms ('stringent quality checks', 'enhanced data screening'), and the actual raw data or specific sources are not public.

The tokenizer is publicly available via the Hugging Face repository and GitHub. It uses Byte-level Byte Pair Encoding (BPE) with a documented vocabulary size of 151,643 regular tokens. Its efficiency and compression rates across multiple languages are discussed in the technical report, and the tokenizer files are fully inspectable, allowing for verification of claimed language support.

The model's parameter count is clearly stated as 1.54 billion total, with 1.31 billion non-embedding parameters. As a dense model, all parameters are active. Detailed architectural hyper-parameters (28 layers, hidden size of 1536, 12 query heads, 2 KV heads) are publicly available in the model configuration files and technical documentation.

Information regarding the specific compute resources used for Qwen2-1.5B is extremely limited. While the technical report mentions the scale of the data (7T tokens), it does not disclose the total GPU/TPU hours, hardware cluster specifications, or the estimated carbon footprint. Cost estimates are entirely absent from official documentation.

While the model provides scores for standard benchmarks (MMLU, GSM8K, HumanEval, etc.) in its technical report, it lacks a dedicated, easy-to-run evaluation suite or the exact prompts used for every result. Third-party researchers have raised significant concerns regarding data contamination in the Qwen series (specifically in math and reasoning benchmarks), which Alibaba has not formally addressed with a detailed leakage audit for this specific version. This significantly impacts the reliability and reproducibility of the reported scores.

The model consistently identifies itself as part of the Qwen family and correctly references its versioning (Qwen2). It does not exhibit the identity confusion seen in some other open-weights models that claim to be GPT-4. Its capabilities and limitations are generally aligned with its 1.5B scale, and it does not make deceptive claims about its nature.

Qwen2-1.5B is released under the Apache 2.0 license, which is a standard, permissive open-source license. This is a significant improvement over the previous proprietary 'Qianwen License' used for larger models in the family. The terms for commercial use and derivative works are clear and follow standard Apache 2.0 protocols.

VRAM requirements are well-documented by both the official team and the community. The model card provides guidance on memory usage for inference (approx. 4.6GB for FP16), and the impact of quantization (INT8/INT4) is documented in various deployment guides (e.g., Ollama, vLLM). Scaling behavior for context length is also generally understood, though official documentation on accuracy-quantization tradeoffs is less detailed.

The model uses a clear naming convention (Qwen2-1.5B), but a formal changelog or semantic versioning for weight updates is not strictly maintained in a centralized way. While major releases (Qwen1.5 to Qwen2 to Qwen2.5) are well-documented, minor iterations or silent updates to the weights on Hugging Face can be difficult to track without manual hash verification.