Kimi K2-Instruct

Active Parameters

1T

Context Length

128K

Modality

Text

Architecture

Mixture of Experts

KV Cache Rate

69 KB/tok

License

Modified MIT License

Release Date

11 Jul 2025

Knowledge Cutoff

-

API Pricing (per 1M)

Input: $0.60 · Output: $2.50

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 1938.09 GB

FP16 · 8K context

Base Weights: 1922.25 GB

KV Cache: 13.34 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

16x NVIDIA H200 141GB

1938.1 / 2030.4 GB usable · TP 8 · PP 2

Determine requirements for your specific workload:

Upgrade to Pro

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.48.3

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashMLA

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Additional required packages

Install alongside standard runtime libraries:

pip install flash_attn

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 7.2k · Context: 128K · Vocab: 163.8kx 61 layersRMSNormPre-AttentionMulti-Layer Attention64Q / 64KV headsHead dim: 112+RMSNormPre-FFNSparse MoE FFN (8/384 experts)SwiGLUIntermediate: 2k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#98

BenchmarkScoreRank

Professional Knowledge

MMLU Pro

0.811

27

Software Engineering

SWE-bench Verified

0.53

31

Graduate-Level QA

GPQA

0.751

75

General Text

Text Arena

1418

106

Web Development

WebDev Arena
auto

1322

114

Intelligence Index

Artificial Analysis
auto

0.22

163

General Knowledge

Reference
MMLU

0.895

🥉

3

Rankings

Overall Rank

#98

Coding Rank

#110

About Kimi K2-Instruct

Kimi K2-Instruct is an instruction-tuned 1T-parameter Mixture-of-Experts model from Moonshot AI engineered for agentic problem-solving and software development. It features advanced tool-calling and long-horizon reasoning across a 128K context window.

Specifications

Attention

Attention Structure

Multi-Layer Attention

Attention Heads

64

Key-Value Heads

64

Position Embedding

ROPE

RoPE Theta

50,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

7,168

Number of Layers

61

FFN Intermediate Size (Dense)

2,048

Multi-Token Prediction Heads

0

Tokenizer

Vocabulary Size

163,840

Mixture of Experts

Total Expert Parameters

32.0B

Number of Experts

384

Active Experts

8

Shared Experts

1

FFN Intermediate Size (per Expert)

2,048

Dense Layers Before MoE

1

About Kimi K2

Moonshot AI's Kimi K2 is a Mixture-of-Experts model featuring one trillion total parameters, activating 32 billion per token. Designed for agentic intelligence, it utilizes a sparse architecture with 384 experts and the MuonClip optimizer for training stability, supporting a 128K token context window.


Other Kimi K2 Models
Kimi K2-Instruct - VRAM Requirements, Specs & Benchmarks