ApX logoApX logo

Qwen3 Next 80B A3B

Active Parameters

80B

Context Length

66K

Modality

Reasoning

Architecture

Mixture of Experts

KV Cache Rate

96 KB/tok

License

Apache-2.0

Release Date

1 Feb 2026

Knowledge Cutoff

Jun 2025

API Pricing (per 1M)

Input: $0.09 · Output: $1.10

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 299.41 GB

FP16 · 8K context

Base Weights: 296.16 GB

KV Cache: 0.75 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

8x NVIDIA H100 80GB

299.4 / 576.0 GB usable · TP 8

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.57.0.dev0

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: AbsoluteHidden: 2k · Context: 66K · Vocab: 151.9kx 48 layersRMSNormPre-AttentionMulti-Head Attention16Q / 2KV headsHead dim: 256+RMSNormPre-FFNSparse MoE FFN (10/512 experts)SwiGLUIntermediate: 512+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#141

BenchmarkScoreRank

Professional Knowledge

MMLU Pro

0.806

29

Graduate-Level QA

GPQA

0.729

81

General Text

Text Arena

1399

123

0.17

135

Intelligence Index

Artificial Analysis
auto

0.11

Standard

0.10

244

254

Rankings

Overall Rank

#141

Coding Rank

#147

About Qwen3 Next 80B A3B

Qwen3-Next-80B-A3B is a sparse Mixture-of-Experts model by Alibaba activating 3B parameters via hybrid Gated DeltaNet linear attention. It delivers ultra-fast sequence modeling and complex mathematical reasoning over an extensible 1M context window.

Specifications

Attention

Attention Structure

Multi-Head Attention

Attention Heads

16

Key-Value Heads

2

Attention Head Dimension

256

Position Embedding

Absolute Position Embedding

RoPE Theta

10,000,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

2,048

Number of Layers

48

FFN Intermediate Size (Dense)

512

Tokenizer

Vocabulary Size

151,936

Mixture of Experts

Total Expert Parameters

79.0B

Number of Experts

512

Active Experts

10

FFN Intermediate Size (per Expert)

512

About Qwen 3

The Alibaba Qwen 3 model family comprises dense and Mixture-of-Experts (MoE) architectures, with parameter counts from 0.6B to 235B. Key innovations include a hybrid reasoning system, offering 'thinking' and 'non-thinking' modes for adaptive processing, and support for extensive context windows, enhancing efficiency and scalability.


Other Qwen 3 Models