Qwen3-235B-A22B

The model's architecture is extensively documented in the Qwen3 Technical Report (arXiv:2505.09388). It is a sparse Mixture-of-Experts (MoE) transformer with 235B total parameters and 22B active parameters. Key components are explicitly named: 94 transformer layers, 128 experts with top-2 routing (8 experts activated per token in later variants), Grouped-Query Attention (GQA) with 64 heads for Q and 4 for KV, Rotary Positional Embedding (RoPE), and SwiGLU activation. The report details the pre-training and post-training stages, including the 'thinking' and 'non-thinking' mode integration.

While the scale of the pre-training corpus is disclosed (36 trillion tokens) and its multilingual breadth is stated (119 languages), the specific composition breakdown (e.g., percentage of web, code, books) is not provided. The documentation mentions 'diverse web sources and documents' and the use of Qwen2.5-VL for data extraction, but lacks a detailed categorical distribution or public access to the training data samples.

The tokenizer is publicly available via Hugging Face and documented in the official 'Read the Docs'. It uses Byte Pair Encoding (BPE) with a vocabulary size of 151,646 tokens. The documentation explicitly describes the subword tokenization method and provides code snippets for implementation. It supports the claimed 119 languages and includes specific control tokens for the 'thinking' mode (<think> and </think>).

Alibaba provides clear and consistent parameter counts: 235B total and 22B active. The architectural breakdown is detailed, specifying 94 layers and the expert routing mechanism (top-2 routing among 128 experts). The distinction between dense and sparse parameters is maintained across all official documentation, including the technical report and model cards.

Information regarding training compute is minimal. While the technical report mentions the use of 'extensive training' and 'reinforcement learning compute', it does not disclose the specific number of GPU/TPU hours, the hardware cluster size used for the flagship 235B model, or the total carbon footprint. Environmental impact data and specific training costs are conspicuously absent.

The technical report provides results for standard benchmarks (AIME, LiveCodeBench, MMLU-Pro) and specifies the versions used. Evaluation settings, such as temperature (0.6) and top_p (0.95) for thinking mode, are documented. However, the full evaluation code and the exact prompts for all benchmarks are not fully public, and third-party verification is limited to early leaderboard entries.

The model demonstrates high identity consistency. It is clearly versioned (e.g., Qwen3-235B-A22B-Instruct-2507) and correctly identifies its capabilities, such as the dual 'thinking' and 'non-thinking' modes. Documentation explicitly guides users on how the model identifies itself through chat templates and system prompts.

The model weights and code are released under the Apache 2.0 license, which is a standard, permissive open-source license. This is consistently stated across Hugging Face, GitHub, and the technical report. The license allows for commercial use, modification, and distribution without conflicting proprietary terms.

Hardware requirements are well-documented for various quantization levels. Official and community sources (e.g., localai.computer, vllm-ascend) provide VRAM estimates: ~460GB for FP16, ~230GB for Q8, and ~115GB for Q4. The impact of context length on memory (YaRN for 128K) is also documented, providing clear guidance for deployment on both consumer and datacenter hardware.

Alibaba uses a clear versioning scheme (e.g., the '2507' suffix for July 2025 updates). Changelogs and blog posts document significant shifts, such as the decision to separate 'Instruct' and 'Thinking' models in the July update. While semantic versioning is not strictly followed in the traditional software sense, the date-based versioning provides a verifiable history of model iterations.