ApX logoApX logo

Qwen3-1.7B

Parameters

1.7B

Context Length

33K

Modality

Text

Architecture

Dense

KV Cache Rate

128 KB/tok

License

Apache 2.0

Release Date

29 Apr 2025

Knowledge Cutoff

Dec 2024

API Pricing (per 1M)

Input: $0.11 · Output: $0.42

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 4.29 GB

INT4 · 8K context

Base Weights: 0.79 GB

KV Cache: 1.00 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

4.3 / 21.6 GB usable

1x RTX 3060 12GB

4.3 / 10.8 GB usable

Apple M4 24GB

4.3 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.51.0

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 2k · Context: 33K · Vocab: 151.9kx 32 layersRMSNormPre-AttentionGrouped-Query Attention32Q / 8KV headsHead dim: 128+RMSNormPre-FFNFeed-Forward NetworkSwiGLUIntermediate: 6.1k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#218

BenchmarkScoreRank

Intelligence Index

Artificial Analysis
auto

0.05

Standard

0.05

321

329

Rankings

Overall Rank

#218

Coding Rank

-

About Qwen3-1.7B

Qwen3-1.7B is a high-efficiency dense foundation model from Alibaba Cloud supporting both Thinking and Non-Thinking modes for edge and mobile deployments. It provides low-latency conversational responses and step-by-step logic over an extensible 131K context.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

32

Key-Value Heads

8

Attention Head Dimension

128

Position Embedding

ROPE

RoPE Theta

1,000,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

2,048

Number of Layers

32

FFN Intermediate Size (Dense)

6,144

Tokenizer

Vocabulary Size

151,936

About Qwen 3

The Alibaba Qwen 3 model family comprises dense and Mixture-of-Experts (MoE) architectures, with parameter counts from 0.6B to 235B. Key innovations include a hybrid reasoning system, offering 'thinking' and 'non-thinking' modes for adaptive processing, and support for extensive context windows, enhancing efficiency and scalability.


Other Qwen 3 Models