ApX logoApX logo

DeepSeek-R1 14B

Parameters

14B

Context Length

131K

Modality

Text

Architecture

Dense

KV Cache Rate

45 KB/tok

License

MIT License

Release Date

27 Dec 2024

Knowledge Cutoff

Jul 2024

API Pricing (per 1M)

Self-hosted only

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 15.27 GB

INT4 · 8K context

Base Weights: 6.52 GB

KV Cache: 6.25 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

15.3 / 21.6 GB usable

2x RTX 4090 24GB

15.3 / 43.2 GB usable · TP 2

Apple M4 24GB

15.3 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.43.1

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 5.1k · Context: 131K · Vocab: 152.1kx 40 layersRMSNormPre-AttentionMulti-Layer Attention80Q / 80KV headsHead dim: 64+RMSNormPre-FFNFeed-Forward NetworkSwiGLUIntermediate: 13.8k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#180

BenchmarkScoreRank

Graduate-Level QA

GPQA

0.591

96

Intelligence Index

Artificial Analysis

0.08

261

Rankings

Overall Rank

#180

Coding Rank

-

About DeepSeek-R1 14B

DeepSeek-R1 14B is a dense reasoning model distilled from DeepSeek-R1 onto the Qwen2.5-14B backbone. It delivers competitive multi-step mathematical reasoning and programming synthesis across a 131K token context window.

Specifications

Attention

Attention Structure

Multi-Layer Attention

Attention Heads

80

Key-Value Heads

80

Position Embedding

ROPE

RoPE Theta

1,000,000

Sliding Window Attention

No

Sliding Window Size

131,072

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

5,120

Number of Layers

40

FFN Intermediate Size (Dense)

13,824

Tokenizer

Vocabulary Size

152,064

About DeepSeek-R1

DeepSeek-R1 is a model family developed for logical reasoning tasks. It incorporates a Mixture-of-Experts architecture for computational efficiency and scalability. The family utilizes Multi-Head Latent Attention and employs reinforcement learning in its training, with some variants integrating cold-start data.


Other DeepSeek-R1 Models