ApX logoApX logo

DeepSeek-R1 8B

Parameters

8B

Context Length

64K

Modality

Text

Architecture

Dense

License

MIT License

Release Date

27 Dec 2024

Knowledge Cutoff

-

API Pricing (per 1M)

Self-hosted only

System Requirements

VRAM requirements for different quantization methods and context sizes

1,024 tokens

18.46 GB VRAM

Consumer

1x RTX 4090

24GB VRAM

Datacenter

1x NVIDIA A100

80GB VRAM

Apple Silicon

1x Apple M3 Max

128GB VRAM

64,000 tokens

99.18 GB VRAM

Consumer

5x RTX 4090

24GB VRAM

Datacenter

2x NVIDIA A100

80GB VRAM

Apple Silicon

1x Apple M3 Max

128GB VRAM

Architecture Diagram

Evaluation Benchmarks

Rank

#197

BenchmarkScoreRank

Graduate-Level QA

GPQA

0.49

106

Intelligence Index

Artificial Analysis

0.07

302

Rankings

Overall Rank

#197

Coding Rank

-

About DeepSeek-R1 8B

DeepSeek-R1 8B is a high-efficiency reasoning model distilled from DeepSeek-R1 onto open-weights backbones. It is optimized for low-hallucination logical deduction, function calling, and automated code generation.

Technical Specifications

Attention

Attention Structure

Multi-Layer Attention

Attention Heads

64

Key-Value Heads

64

Position Embedding

ROPE

RoPE Theta

500,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Hidden Dimension Size

4,096

Number of Layers

40

FFN Intermediate Size (Dense)

14,336

Tokenizer

Vocabulary Size

128,256

About DeepSeek-R1

DeepSeek-R1 is a model family developed for logical reasoning tasks. It incorporates a Mixture-of-Experts architecture for computational efficiency and scalability. The family utilizes Multi-Head Latent Attention and employs reinforcement learning in its training, with some variants integrating cold-start data.


Other DeepSeek-R1 Models