Active Parameters
48B
Context Length
1.05M
Modality
Text
Architecture
Mixture of Experts
KV Cache Rate
10 KB/tok
License
MIT
Release Date
1 Nov 2025
Knowledge Cutoff
Oct 2024
API Pricing (per 1M)
Self-hosted only
Precision
Context
Workload Memory: 26.33 GB
INT4 · 8K context
Base Weights: 23.75 GB
KV Cache: 0.08 GB
Runtime & Graphs: 2.50 GB
1x NVIDIA A40 48GB
26.3 / 43.2 GB usable
2x RTX 4090 24GB
26.3 / 43.2 GB usable · TP 2
Apple M3 Pro 36GB
26.3 / 27.0 GB usable
Determine requirements for your specific workload:
Open in VRAM Calculator
Checkpoint format
FP16
CUDA
>= 11.8
PyTorch
>= 2.1
Transformers
>= 4.57.1
| Architecture | Precision | Attention kernel |
|---|---|---|
Blackwell, Hopper | Native FP16 | FlashAttention-3 |
Ada Lovelace, Ampere | Native FP16 | FlashAttention-2 |
Additional required packages
Install alongside standard runtime libraries:
pip install einops fla
Rank
#207
| Benchmark | Score | Rank |
|---|---|---|
Intelligence Index | 0.07 | 277 |
Overall Rank
#207
Coding Rank
-
Kimi Linear 48B A3B is an open-source hybrid linear attention MoE model by Moonshot AI activating 3B parameters per token. Powered by Kimi Delta Attention, it achieves high decoding throughput and a 75% reduction in KV cache across a 1M context window.
Attention
Attention Structure
Multi-Head Attention
Attention Heads
32
Key-Value Heads
1
Attention Head Dimension
72
Position Embedding
Absolute Position Embedding
RoPE Theta
10,000
Normalization
RMS Normalization
Activation Function
SwigLU
Dimensions
Base Precision
FP16
Hidden Dimension Size
4,096
Number of Layers
36
FFN Intermediate Size (Dense)
1,024
Multi-Token Prediction Heads
0
Tokenizer
Vocabulary Size
163,840
Mixture of Experts
Total Expert Parameters
3.0B
Number of Experts
128
Active Experts
8
Shared Experts
1
FFN Intermediate Size (per Expert)
1,024
Dense Layers Before MoE
1
Moonshot AI's hybrid linear attention architecture with Kimi Delta Attention for efficient long-context processing.