ApX logoApX logo

Qwen2.5-32B

Parameters

32B

Context Length

131K

Modality

Text

Architecture

Dense

KV Cache Rate

159 KB/tok

License

Apache 2.0

Release Date

19 Sept 2024

Knowledge Cutoff

Jun 2024

API Pricing (per 1M)

Input: $0.87 · Output: $0.87

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 18.65 GB

INT4 · 8K context

Base Weights: 14.90 GB

KV Cache: 1.25 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

18.6 / 21.6 GB usable

2x RTX 4090 24GB

18.6 / 43.2 GB usable · TP 2

Apple M3 Pro 36GB

18.6 / 27.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.43.1

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 8.2k · Context: 131K · Vocab: 152.1kx 60 layersRMSNormPre-AttentionGrouped-Query Attention96Q / 8KV headsHead dim: 85+RMSNormPre-FFNFeed-Forward NetworkSwiGLUIntermediate: 27.6k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#180

BenchmarkScoreRank

Software Engineering

SWE-bench Verified

0.38

34

Professional Knowledge

MMLU Pro

0.69

45

Graduate-Level QA

GPQA

0.495

104

Intelligence Index

Artificial Analysis

0.07

285

General Knowledge

Reference
MMLU

0.833

16

Coding

Archived
Aider Coding

0.16

19

Rankings

Overall Rank

#180

Coding Rank

#144

About Qwen2.5-32B

Qwen2.5-32B is a high-capacity dense transformer from Alibaba Cloud engineered for complex reasoning, long-form content creation, and JSON data parsing. It delivers high stability across diverse system prompts over a 131K token context window.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

96

Key-Value Heads

8

Position Embedding

ROPE

RoPE Theta

1,000,000

Sliding Window Attention

No

Sliding Window Size

131,072

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

8,192

Number of Layers

60

FFN Intermediate Size (Dense)

27,648

Tokenizer

Vocabulary Size

152,064

About Qwen2.5

Qwen2.5 by Alibaba is a family of dense, decoder-only language models available in various sizes, with some variants utilizing Mixture-of-Experts. These models are pretrained on large-scale datasets, supporting extended context lengths and multilingual communication. The family includes specialized models for coding, mathematics, and multimodal tasks, such as vision and audio processing.


Other Qwen2.5 Models