Parameters
4B
Context Length
262K
Modality
Multimodal
Auxiliary Parameters
350M
Architecture
Dense
KV Cache Rate
128 KB/tok
License
Apache 2.0
Release Date
24 Feb 2026
Knowledge Cutoff
-
API Pricing (per 1M)
Input: $0.03 · Output: $0.15
Precision
Context
Workload Memory: 5.36 GB
INT4 · 8K context
Base Weights: 1.86 GB
KV Cache: 1.00 GB
Runtime & Graphs: 2.50 GB
1x NVIDIA L4 24GB
5.4 / 21.6 GB usable
1x RTX 3060 12GB
5.4 / 10.8 GB usable
Apple M4 24GB
5.4 / 18.0 GB usable
Determine requirements for your specific workload:
Open in VRAM Calculator
Checkpoint format
FP16
CUDA
>= 11.8
PyTorch
>= 2.1
Transformers
>= 4.57.0.dev0
| Architecture | Precision | Attention kernel |
|---|---|---|
Blackwell, Hopper | Native FP16 | FlashAttention-3 |
Ada Lovelace, Ampere | Native FP16 | FlashAttention-2 |
Rank
#145
| Benchmark | Score | Rank |
|---|---|---|
Professional Knowledge | 0.791 | 32 |
Graduate-Level QA | 0.762 | 72 |
Coding Index | 0.20 | 133 |
Intelligence Index | auto 0.13 Standard 0.11 | 234 258 |
Overall Rank
#145
Coding Rank
#149
Qwen3.5-4B is Alibaba Cloud's compact multimodal foundation model with 4B parameters, released February 2026. It uses a hybrid architecture combining Gated Delta Networks and Gated Attention in an 8×(3×DeltaNet→FFN→1×Attention→FFN) pattern. It achieves MMLU-Pro (79.1%), GPQA Diamond (76.2%), HMMT benchmarks (74%/77%), and strong vision-language scores. Features unified vision-language capabilities, 262k native context (extensible to 1M), multi-token prediction training, and delivers efficient performance across reasoning, coding, multimodal understanding, and multilingual tasks covering 201 languages.
Attention
Attention Structure
Grouped-Query Attention
Attention Heads
16
Key-Value Heads
4
Attention Head Dimension
256
Position Embedding
ROPE
RoPE Theta
10,000,000
Linear Attention
Yes
Linear Attention Ratio
75.0%
Normalization
RMS Normalization
Activation Function
SwigLU
Dimensions
Base Precision
FP16
Auxiliary Parameters
350M
Hidden Dimension Size
2,560
Number of Layers
32
FFN Intermediate Size (Dense)
9,216
Multi-Token Prediction Heads
1
Tokenizer
Vocabulary Size
248,320
Qwen 3.5 is Alibaba Cloud's latest-generation foundation model family, released February 2026. It represents a significant leap forward, integrating breakthroughs in multimodal learning (unified vision-language foundation), efficient hybrid architecture (Gated Delta Networks with sparse Mixture-of-Experts), scalable reinforcement learning across million-agent environments, and global linguistic coverage spanning 201 languages. Available under Apache 2.0 license with open weights.