ApX logoApX logo

Kimi Linear 48B A3B Instruct

Active Parameters

48B

Context Length

1.05M

Modality

Text

Architecture

Mixture of Experts

KV Cache Rate

10 KB/tok

License

MIT

Release Date

1 Nov 2025

Knowledge Cutoff

Oct 2024

API Pricing (per 1M)

Self-hosted only

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 26.33 GB

INT4 · 8K context

Base Weights: 23.75 GB

KV Cache: 0.08 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA A40 48GB

26.3 / 43.2 GB usable

2x RTX 4090 24GB

26.3 / 43.2 GB usable · TP 2

Apple M3 Pro 36GB

26.3 / 27.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.57.1

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Additional required packages

Install alongside standard runtime libraries:

pip install einops fla

Architecture Diagram

Input TokensToken EmbeddingPosition: AbsoluteHidden: 4.1k · Context: 1.05M · Vocab: 163.8kx 36 layersRMSNormPre-AttentionMulti-Head Attention32Q / 1KV headsHead dim: 72+RMSNormPre-FFNSparse MoE FFN (8/128 experts)SwiGLUIntermediate: 1k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#207

BenchmarkScoreRank

Intelligence Index

Artificial Analysis

0.07

277

Rankings

Overall Rank

#207

Coding Rank

-

About Kimi Linear 48B A3B Instruct

Kimi Linear 48B A3B is an open-source hybrid linear attention MoE model by Moonshot AI activating 3B parameters per token. Powered by Kimi Delta Attention, it achieves high decoding throughput and a 75% reduction in KV cache across a 1M context window.

Specifications

Attention

Attention Structure

Multi-Head Attention

Attention Heads

32

Key-Value Heads

1

Attention Head Dimension

72

Position Embedding

Absolute Position Embedding

RoPE Theta

10,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

4,096

Number of Layers

36

FFN Intermediate Size (Dense)

1,024

Multi-Token Prediction Heads

0

Tokenizer

Vocabulary Size

163,840

Mixture of Experts

Total Expert Parameters

3.0B

Number of Experts

128

Active Experts

8

Shared Experts

1

FFN Intermediate Size (per Expert)

1,024

Dense Layers Before MoE

1

About Kimi Linear

Moonshot AI's hybrid linear attention architecture with Kimi Delta Attention for efficient long-context processing.


Other Kimi Linear Models
  • No related models available