Qwen3-30B-A3B

The model's architecture is extensively documented in the Qwen3 Technical Report (arXiv:2505.09388). It is a Mixture-of-Experts (MoE) transformer with 48 layers, utilizing Grouped Query Attention (GQA) with 32 query heads and 4 KV heads. The MoE design features 128 experts with 8 active per token. Key technical components like SwiGLU activation, RoPE (Rotary Position Embeddings), and RMSNorm are explicitly detailed. The report also describes a unique 'thinking mode' hybrid system and a three-stage pre-training methodology (General, Reasoning, and Long-context).

Alibaba discloses that the model was trained on a 36 trillion token corpus spanning 119 languages. While the general categories of data are mentioned—including web data, books, PDFs, and synthetic data generated by previous Qwen models (Qwen2.5-VL for extraction, Qwen2.5-Math/Coder for synthetic generation)—there is no precise percentage breakdown of the dataset composition (e.g., exact ratios of code vs. web vs. books). The filtering and cleaning methodologies are described at a high level but lack granular technical specifics.

The tokenizer is publicly available via the Hugging Face repository and official Qwen GitHub. It uses Byte Pair Encoding (BPE) with a vocabulary size of 151,936 tokens. It supports 119 languages and dialects, which is verified by the model's extensive multilingual benchmark performance. Documentation provides clear instructions for handling special tokens, including the <think> tags used in reasoning mode.

The model provides exemplary transparency regarding its parameter count. It explicitly states a total of 30.5 billion parameters, with a non-embedding parameter count of 29.9 billion. Crucially for an MoE model, it clearly discloses that only 3.3 billion parameters are active per token during inference. The architectural breakdown of 128 total experts and 8 activated experts is consistently reported across all official documentation.

While the technical report mentions the use of scaling laws to tune hyperparameters and the scale of the training (36T tokens), it lacks specific details on the total GPU/TPU hours consumed, the specific hardware clusters used for the full training run, or the estimated carbon footprint. The information provided is limited to the scale of the data rather than the specific compute resources utilized.

The technical report provides results for standard benchmarks such as MMLU (81.38), GSM8K, MATH, and LiveCodeBench. Evaluation settings, including sampling parameters (Temperature=0.6, TopP=0.95 for thinking mode) and prompt templates, are documented. However, the full evaluation code and the exact internal test sets used for 'thinking mode' validation are not fully public, and third-party verification for the newest Qwen3 variants is still emerging.

The model demonstrates high identity consistency, correctly identifying itself as a Qwen3 series model in both its system prompts and documentation. It maintains clear versioning between the base, instruct, and 'thinking' variants. There are no documented cases of the model claiming to be a competitor's product, and it is transparent about its dual-mode (thinking vs. non-thinking) capabilities.

The model is released under the Apache 2.0 license, which is a standard, permissive open-source license. This allows for commercial use, modification, and distribution without the restrictive 'custom' terms often found in other 'open' weights models. The licensing is consistent across the weights on Hugging Face and the official GitHub repository.

Hardware requirements are well-documented for various deployment scenarios. Official documentation specifies VRAM needs for standard inference and provides guidance for ultra-long context (up to 1M tokens requiring ~240GB VRAM). It also notes the memory savings (approx. 10GB) when disabling specific multimodal components. Quantization support is mentioned for frameworks like llama.cpp and vLLM, though detailed accuracy-tradeoff curves for all quantization levels are not fully provided.

Alibaba uses a date-based versioning system (e.g., 2507 for July 2025 updates) and maintains a clear distinction between 'Base', 'Instruct', and 'Thinking' versions. While changelogs are provided on GitHub and Hugging Face, the frequency of 'silent' updates to the underlying API endpoints without corresponding weight version bumps remains a minor concern for long-term reproducibility.