ApX logoApX logo

DeepSeek-R1 70B

Parameters

70B

Context Length

33K

Modality

Text

Architecture

Dense

KV Cache Rate

90 KB/tok

License

MIT License

Release Date

27 Dec 2024

Knowledge Cutoff

-

API Pricing (per 1M)

Input: $0.70 · Output: $1.10

Hardware Sizing

Precision

Context

Workload Memory: 70.10 GB

INT4 · 8K context

Base Weights: 32.60 GB

KV Cache: 35.00 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA H100 80GB

70.1 / 72.0 GB usable

4x RTX 4090 24GB

70.1 / 86.4 GB usable · TP 4

Apple M4 Max 128GB

70.1 / 96.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 8.2k · Context: 33K · Vocab: 128.3kx 80 layersRMSNormPre-AttentionMulti-Layer Attention112Q / 112KV headsHead dim: 128+RMSNormPre-FFNFeed-Forward NetworkSwishIntermediate: 28.7k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#162

BenchmarkScoreRank

Graduate-Level QA

GPQA

0.652

92

Intelligence Index

Artificial Analysis

0.08

254

Rankings

Overall Rank

#162

Coding Rank

-

About DeepSeek-R1 70B

DeepSeek-R1 70B is a dense reasoning foundation model distilled from DeepSeek-R1 onto the Llama-3.3-70B-Instruct architecture. It combines deep logical inference and competitive programming skills with efficient enterprise deployment profiles.

Technical Specifications

Attention

Attention Structure

Multi-Layer Attention

Attention Heads

112

Key-Value Heads

112

Attention Head Dimension

128

Position Embedding

ROPE

RoPE Theta

500,000

Normalization

RMS Normalization

Activation Function

Swish

Dimensions

Hidden Dimension Size

8,192

Number of Layers

80

FFN Intermediate Size (Dense)

28,672

Tokenizer

Vocabulary Size

128,256

About DeepSeek-R1

DeepSeek-R1 is a model family developed for logical reasoning tasks. It incorporates a Mixture-of-Experts architecture for computational efficiency and scalability. The family utilizes Multi-Head Latent Attention and employs reinforcement learning in its training, with some variants integrating cold-start data.


Other DeepSeek-R1 Models