Qwen3.5-4B

Parameters

4B

Context Length

262K

Modality

Multimodal

Auxiliary Parameters

350M

Architecture

Dense

KV Cache Rate

128 KB/tok

License

Apache 2.0

Release Date

24 Feb 2026

Knowledge Cutoff

-

API Pricing (per 1M)

Input: $0.03 · Output: $0.15

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 5.36 GB

INT4 · 8K context

Base Weights: 1.86 GB

KV Cache: 1.00 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

5.4 / 21.6 GB usable

1x RTX 3060 12GB

5.4 / 10.8 GB usable

Apple M4 24GB

5.4 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.57.0.dev0

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 2.6k · Context: 262K · Vocab: 248.3kx 32 layersRMSNormPre-AttentionGrouped-Query Attention16Q / 4KV headsHead dim: 256+RMSNormPre-FFNFeed-Forward NetworkSwiGLUIntermediate: 9.2k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#145

BenchmarkScoreRank

Professional Knowledge

MMLU Pro

0.791

32

Graduate-Level QA

GPQA

0.762

72

0.20

133

Intelligence Index

Artificial Analysis
auto

0.13

Standard

0.11

234

258

Rankings

Overall Rank

#145

Coding Rank

#149

About Qwen3.5-4B

Qwen3.5-4B is Alibaba Cloud's compact multimodal foundation model with 4B parameters, released February 2026. It uses a hybrid architecture combining Gated Delta Networks and Gated Attention in an 8×(3×DeltaNet→FFN→1×Attention→FFN) pattern. It achieves MMLU-Pro (79.1%), GPQA Diamond (76.2%), HMMT benchmarks (74%/77%), and strong vision-language scores. Features unified vision-language capabilities, 262k native context (extensible to 1M), multi-token prediction training, and delivers efficient performance across reasoning, coding, multimodal understanding, and multilingual tasks covering 201 languages.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

16

Key-Value Heads

4

Attention Head Dimension

256

Position Embedding

ROPE

RoPE Theta

10,000,000

Linear Attention

Yes

Linear Attention Ratio

75.0%

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Auxiliary Parameters

350M

Hidden Dimension Size

2,560

Number of Layers

32

FFN Intermediate Size (Dense)

9,216

Multi-Token Prediction Heads

1

Tokenizer

Vocabulary Size

248,320

About Qwen 3.5

Qwen 3.5 is Alibaba Cloud's latest-generation foundation model family, released February 2026. It represents a significant leap forward, integrating breakthroughs in multimodal learning (unified vision-language foundation), efficient hybrid architecture (Gated Delta Networks with sparse Mixture-of-Experts), scalable reinforcement learning across million-agent environments, and global linguistic coverage spanning 201 languages. Available under Apache 2.0 license with open weights.


Other Qwen 3.5 Models