Qwen3-4B

The model's architecture is extensively documented in the Qwen3 Technical Report (arXiv:2505.09388). It is a dense causal language model with 36 layers, utilizing Grouped Query Attention (GQA) with 32 query heads and 8 KV heads. Key technical components such as SwiGLU activations, RMSNorm with pre-normalization, and Rotary Position Embeddings (RoPE) are explicitly detailed. A significant innovation, the 'dual-mode' thinking/non-thinking architecture, is described as a unified framework that allows dynamic switching between reasoning states. The training methodology is clearly outlined as a three-stage pipeline (General, Reasoning, and Long Context stages).

Alibaba discloses that the model was trained on a massive 36 trillion token dataset covering 119 languages. The documentation provides a high-level breakdown of data types, including web data, books, STEM, coding, and synthetic data generated by previous Qwen models (Qwen2.5-Math/Coder). The three-stage data curriculum is well-defined, showing how the data mix evolved from general knowledge to reasoning-intensive content. However, specific percentage breakdowns of the 36T tokens (e.g., exact ratios of web vs. books) and the specific identities of the non-web datasets are not fully disclosed, which is a common gap in large-scale model transparency.

The tokenizer is publicly available via the official GitHub repository and Hugging Face. It uses byte-level byte-pair encoding (BBPE) with a large vocabulary size of 151,669 tokens, which is consistent across the Qwen series. The documentation confirms support for 119 languages, and the tokenizer's alignment with this claim is verifiable through the provided configuration files. The use of 'tied embeddings' for the 4B variant is also explicitly documented to optimize memory efficiency.

The parameter count is precisely stated as 4.0 billion total parameters, with a non-embedding parameter count of 3.6 billion. As a dense model, all parameters are active during inference, which is clearly distinguished from the MoE variants in the same family (e.g., Qwen3-30B-A3B). The architectural breakdown, including the number of layers (36) and attention head configurations, is fully transparent in the technical report.

While the technical report describes the training stages and the scale of the data (36T tokens), it conspicuously lacks specific details regarding the compute resources used. There is no disclosure of total GPU/TPU hours, the specific hardware clusters utilized for the 4B variant, or the estimated carbon footprint. This information is treated as proprietary, which is a significant transparency gap according to the scoring guidelines.

The model is evaluated on a wide array of standard benchmarks (MMLU, GSM8K, HumanEval, etc.) with results published in the technical report. Evaluation settings, such as the use of 'thinking mode' for reasoning tasks and specific sampling parameters (Temperature=0.6, TopP=0.95), are provided. The team recommends standardized prompts for benchmarking, such as 'Please reason step by step'. While the full evaluation codebase is not a single-click reproduction script, the level of detail regarding prompts and modes is significantly higher than industry averages.

The model demonstrates high identity consistency, correctly identifying itself as part of the Qwen3 family. It is transparent about its dual-mode capabilities, and the system is designed to explicitly signal its state (e.g., using <think> tags). There are no documented instances of the model claiming to be a competitor's product or misrepresenting its 4B scale.

The model weights and associated code are released under the Apache 2.0 license, which is a highly permissive, standard open-source license. This allows for commercial use, modification, and distribution without the 'open-weights but restricted' ambiguity found in some other model families. The licensing terms are clearly stated on Hugging Face and the official GitHub repository.

Hardware requirements are well-documented for various deployment scenarios. Official documentation and third-party sources provide VRAM estimates for FP16 (approx. 8-10GB) and quantized versions (4-bit requiring ~2-4GB). The impact of context length on memory is addressed, with native support for 32K tokens and scaling up to 131K via YaRN. Quantization support is verified through the availability of GGUF and other formats.

The model follows a clear versioning path (Qwen3-4B vs. updated versions like Qwen3-4B-Instruct-2507). A GitHub repository is maintained to track issues and updates. However, while version numbers are present, a detailed, granular changelog documenting specific weight drifts or performance changes between minor iterations is less comprehensive than a full semantic versioning system.