DeepSeek-R1 671B

Active Parameters

671B

Context Length

131K

Modality

Text

Architecture

Mixture of Experts

KV Cache Rate

69 KB/tok

License

MIT License

Release Date

27 Dec 2024

Knowledge Cutoff

Jul 2024

API Pricing (per 1M)

Input: $0.50 · Output: $2.15

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 1325.07 GB

FP16 · 8K context

Base Weights: 1318.75 GB

KV Cache: 3.81 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

16x NVIDIA H200 141GB

1325.1 / 2030.4 GB usable · TP 8 · PP 2

Determine requirements for your specific workload:

Upgrade to Pro

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.46.3

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashMLA

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Additional required packages

Install alongside standard runtime libraries:

pip install flash_attn

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 2k · Context: 131K · Vocab: 129.3kx 61 layersRMSNormPre-AttentionMulti-Layer Attention128Q / 128KV headsHead dim: 16+RMSNormPre-FFNSparse MoE FFN (6/64 experts)SwishIntermediate: 2k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#135

BenchmarkScoreRank

Professional Knowledge

MMLU Pro

0.85

13

0.524

21

Graduate-Level QA

GPQA

0.81

62

General Text

Text Arena

1422

102

Agentic Index

Artificial Analysis

0.01

113

0.25

124

Intelligence Index

Artificial Analysis

0.13

233

StackEval

Archived
ProLLM Stack Eval

0.956

5

QA Assistant

Archived
ProLLM QA Assistant

0.964

5

Coding

Archived
Aider Coding

0.71

7

Summarization

Archived
ProLLM Summarization

0.774

14

Rankings

Overall Rank

#135

Coding Rank

#119

About DeepSeek-R1 671B

DeepSeek-R1 is DeepSeek's flagship 671B Mixture-of-Experts reasoning model engineered with large-scale reinforcement learning for complex problem-solving. It generates explicit multi-step chains of thought for state-of-the-art math, coding, and scientific deduction.

Specifications

Attention

Attention Structure

Multi-Layer Attention

Attention Heads

128

Key-Value Heads

128

Position Embedding

ROPE

RoPE Theta

10,000

Normalization

RMS Normalization

Activation Function

Swish

Dimensions

Base Precision

FP16

Hidden Dimension Size

2,048

Number of Layers

61

FFN Intermediate Size (Dense)

2,048

Multi-Token Prediction Heads

1

Tokenizer

Vocabulary Size

129,280

Mixture of Experts

Total Expert Parameters

37.0B

Number of Experts

64

Active Experts

6

Shared Experts

1

FFN Intermediate Size (per Expert)

2,048

Dense Layers Before MoE

3

About DeepSeek-R1

DeepSeek-R1 is a model family developed for logical reasoning tasks. It incorporates a Mixture-of-Experts architecture for computational efficiency and scalability. The family utilizes Multi-Head Latent Attention and employs reinforcement learning in its training, with some variants integrating cold-start data.


Other DeepSeek-R1 Models