ApX logoApX logo

DeepSeek-V3.2

Active Parameters

671B

Context Length

128K

Modality

Text

Architecture

Mixture of Experts

KV Cache Rate

13 KB/tok

License

MIT

Release Date

10 Jan 2026

Knowledge Cutoff

May 2025

API Pricing (per 1M)

Input: $0.26 · Output: $0.38

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 1321.36 GB

FP16 · 8K context

Base Weights: 1318.75 GB

KV Cache: 0.10 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

16x NVIDIA H200 141GB

1321.4 / 2030.4 GB usable · TP 8 · PP 2

Determine requirements for your specific workload:

Upgrade to Pro

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.46.3

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashMLA

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: AbsoluteHidden: 7.2k · Context: 128K · Vocab: 129.3kx 61 layersRMSNormPre-AttentionDeepSeek Sparse Attention128Q / 1KV headsHead dim: 56+RMSNormPre-FFNSparse MoE FFN (9/257 experts)SwiGLUIntermediate: 2k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#100

BenchmarkScoreRank

Professional Knowledge

MMLU Pro

0.85

13

Software Engineering

SWE-bench Verified

0.70

15

Graduate-Level QA

GPQA

0.824

56

Web Development

WebDev Arena
high

1362

Standard

1326

97

109

General Text

Text Arena
auto

1423

Standard

1425

100

97

0.44

99

Intelligence Index

Artificial Analysis
auto

0.21

Standard

0.16

154

207

Coding

Archived
Aider Coding

0.74

5

Rankings

Overall Rank

#100

Coding Rank

#98

About DeepSeek-V3.2

DeepSeek-V3.2 is an advanced 671B Mixture-of-Experts model from DeepSeek optimized for autonomous agents and complex reasoning. Featuring DeepSeek Sparse Attention and integrated thinking modes, it supports long-horizon execution across a 164K context window.

Specifications

Attention

Attention Structure

DeepSeek Sparse Attention

Attention Heads

128

Key-Value Heads

1

Position Embedding

Absolute Position Embedding

RoPE Theta

10,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

7,168

Number of Layers

61

FFN Intermediate Size (Dense)

2,048

Multi-Token Prediction Heads

1

Tokenizer

Vocabulary Size

129,280

Mixture of Experts

Total Expert Parameters

37.0B

Number of Experts

257

Active Experts

9

Shared Experts

1

FFN Intermediate Size (per Expert)

2,048

Dense Layers Before MoE

3

About DeepSeek-V3

DeepSeek-V3 is a Mixture-of-Experts (MoE) language model comprising 671B parameters with 37B activated per token. Its architecture incorporates Multi-head Latent Attention and DeepSeekMoE for efficient inference and training. Innovations include an auxiliary-loss-free load balancing strategy and a multi-token prediction objective, trained on 14.8T tokens.


Other DeepSeek-V3 Models