ApX logoApX logo

Qwen3-235B-A22B

Active Parameters

235B

Context Length

131K

Modality

Text

Architecture

Mixture of Experts

KV Cache Rate

400 KB/tok

License

Apache 2.0

Release Date

29 Apr 2025

Knowledge Cutoff

-

API Pricing (per 1M)

Input: $0.26 · Output: $1.06

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 484.32 GB

FP16 · 8K context

Base Weights: 478.70 GB

KV Cache: 3.13 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

8x NVIDIA H100 80GB

484.3 / 576.0 GB usable · TP 8

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.51.0

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 10.2k · Context: 131K · Vocab: 151.9kx 100 layersRMSNormPre-AttentionGrouped-Query Attention128Q / 8KV headsHead dim: 128+RMSNormPre-FFNSparse MoE FFN (8/128 experts)SwiGLUIntermediate: 1.5k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#150

BenchmarkScoreRank

Professional Knowledge

MMLU Pro

0.682

47

Graduate-Level QA

GPQA

0.775

71

General Text

Text Arena

1422

101

Agentic Index

Artificial Analysis

0.01

108

0.22

129

Intelligence Index

Artificial Analysis

0.12

239

General Knowledge

Reference
MMLU

0.878

6

Coding

Archived
Aider Coding

0.60

10

Rankings

Overall Rank

#150

Coding Rank

#140

About Qwen3-235B-A22B

Qwen3-235B-A22B is Alibaba Cloud's flagship open-source Mixture-of-Experts model activating 22B parameters for advanced computational linguistics and STEM problem solving. It combines dynamic thinking modes with sparse MoE efficiency across 119 languages.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

128

Key-Value Heads

8

Attention Head Dimension

128

Position Embedding

ROPE

RoPE Theta

1,000,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

10,240

Number of Layers

100

FFN Intermediate Size (Dense)

1,536

Tokenizer

Vocabulary Size

151,936

Mixture of Experts

Total Expert Parameters

22.0B

Number of Experts

128

Active Experts

8

FFN Intermediate Size (per Expert)

1,536

About Qwen 3

The Alibaba Qwen 3 model family comprises dense and Mixture-of-Experts (MoE) architectures, with parameter counts from 0.6B to 235B. Key innovations include a hybrid reasoning system, offering 'thinking' and 'non-thinking' modes for adaptive processing, and support for extensive context windows, enhancing efficiency and scalability.


Other Qwen 3 Models