ApX logoApX logo

Qwen2.5-3B

Parameters

3B

Context Length

33K

Modality

Text

Architecture

Dense

KV Cache Rate

54 KB/tok

License

Qwen Research License Agreement

Release Date

19 Sept 2024

Knowledge Cutoff

-

API Pricing (per 1M)

Self-hosted only

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 4.32 GB

INT4 · 8K context

Base Weights: 1.40 GB

KV Cache: 0.42 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

4.3 / 21.6 GB usable

1x RTX 3060 12GB

4.3 / 10.8 GB usable

Apple M4 24GB

4.3 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.40.1

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 2.3k · Context: 33K · Vocab: 151.9kx 36 layersRMSNormPre-AttentionGrouped-Query Attention48Q / 8KV headsHead dim: 48+RMSNormPre-FFNFeed-Forward NetworkSwiGLUIntermediate: 11k+Final RMSNormOutput Logits

Evaluation Benchmarks

No evaluation benchmarks for Qwen2.5-3B available.

Rankings

Overall Rank

-

Coding Rank

-

About Qwen2.5-3B

Qwen2.5-3B is an efficient dense foundation model developed by Alibaba Cloud balancing compact resource requirements with strong coding and math skills. It offers robust instruction adherence and multilingual fluency over an extensible 128K context window.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

48

Key-Value Heads

8

Position Embedding

ROPE

RoPE Theta

1,000,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

2,304

Number of Layers

36

FFN Intermediate Size (Dense)

11,008

Tokenizer

Vocabulary Size

151,936

About Qwen2.5

Qwen2.5 by Alibaba is a family of dense, decoder-only language models available in various sizes, with some variants utilizing Mixture-of-Experts. These models are pretrained on large-scale datasets, supporting extended context lengths and multilingual communication. The family includes specialized models for coding, mathematics, and multimodal tasks, such as vision and audio processing.


Other Qwen2.5 Models