ApX logoApX logo

DeepSeek-R1 32B

Parameters

32B

Context Length

131K

Modality

Text

Architecture

Dense

License

MIT License

Release Date

27 Dec 2024

Knowledge Cutoff

Jul 2024

API Pricing (per 1M)

Self-hosted only

System Requirements

VRAM requirements for different quantization methods and context sizes

1,024 tokens

71.87 GB VRAM

Consumer

4x RTX 4090

24GB VRAM

Datacenter

1x NVIDIA A100

80GB VRAM

Apple Silicon

1x Apple M3 Max

128GB VRAM

131,072 tokens

474.57 GB VRAM

Consumer

25x RTX 4090

24GB VRAM

Datacenter

7x NVIDIA A100

80GB VRAM

Apple Silicon

5x Apple M3 Max

128GB VRAM

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 8.2k · Context: 131K · Vocab: 152.1kx 60 layersRMSNormPre-AttentionMulti-Layer Attention96Q / 96KV headsHead dim: 85+RMSNormPre-FFNFeed-Forward NetworkSwishIntermediate: 27.6k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#153

BenchmarkScoreRank

Graduate-Level QA

GPQA

0.621

95

Intelligence Index

Artificial Analysis

0.08

204

Rankings

Overall Rank

#153

Coding Rank

-

About DeepSeek-R1 32B

DeepSeek-R1 32B is a high-capacity dense reasoning model distilled from DeepSeek-R1 onto the Qwen2.5-32B architecture. It offers frontier-level performance on complex coding, mathematical proofs, and analytical tasks with a 131K context window.

Technical Specifications

Attention

Attention Structure

Multi-Layer Attention

Attention Heads

96

Key-Value Heads

96

Attention Head Dimension

-

Position Embedding

ROPE

RoPE Theta

1,000,000

Sliding Window Attention

No

Sliding Window Size

131,072

Sliding Window Ratio

-

Linear Attention

-

Linear Attention Ratio

-

Normalization

RMS Normalization

Activation Function

Swish

Dimensions

Auxiliary Parameters

-

Hidden Dimension Size

8,192

Number of Layers

60

FFN Intermediate Size (Dense)

27,648

Multi-Token Prediction Heads

-

Tokenizer

Vocabulary Size

152,064

About DeepSeek-R1

DeepSeek-R1 is a model family developed for logical reasoning tasks. It incorporates a Mixture-of-Experts architecture for computational efficiency and scalability. The family utilizes Multi-Head Latent Attention and employs reinforcement learning in its training, with some variants integrating cold-start data.


Other DeepSeek-R1 Models