ApX logoApX logo

Qwen2-1.5B

Parameters

1.5B

Context Length

33K

Modality

Text

Architecture

Dense

KV Cache Rate

36 KB/tok

License

Apache 2.0

Release Date

7 Jun 2024

Knowledge Cutoff

Dec 2023

API Pricing (per 1M)

Self-hosted only

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 3.48 GB

INT4 · 8K context

Base Weights: 0.70 GB

KV Cache: 0.28 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

3.5 / 21.6 GB usable

1x RTX 3060 12GB

3.5 / 10.8 GB usable

Apple M4 24GB

3.5 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

BF16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.40.1

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native BF16

FlashAttention-3

Ada Lovelace, Ampere

Native BF16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 1.5k · Context: 33K · Vocab: 151.9kx 24 layersRMSNormPre-AttentionGrouped-Query Attention32Q / 8KV headsHead dim: 48+RMSNormPre-FFNFeed-Forward NetworkSwiGLUIntermediate: 9k+Final RMSNormOutput Logits

Evaluation Benchmarks

No evaluation benchmarks for Qwen2-1.5B available.

Rankings

Overall Rank

-

Coding Rank

-

About Qwen2-1.5B

Qwen2-1.5B is a compact, open-weights causal transformer developed by Alibaba Cloud for efficient on-device text generation and lightweight NLP tasks. It features Grouped-Query Attention and broad multilingual support across a 32K context window.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

32

Key-Value Heads

8

Position Embedding

ROPE

RoPE Theta

1,000,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

BF16

Hidden Dimension Size

1,536

Number of Layers

24

FFN Intermediate Size (Dense)

8,960

Tokenizer

Vocabulary Size

151,936

About Qwen2

The Alibaba Qwen2 model family comprises large language models built upon the Transformer architecture. It includes both dense and Mixture-of-Experts (MoE) variants, designed for diverse language tasks. Technical features include Grouped Query Attention and support for extended context lengths up to 131,072 tokens, optimizing memory footprint for inference.


Other Qwen2 Models