Kimi Linear 48B A3B Instruct
Active Parameters
48B
Context Length
1.05M
Modality
Text
Architecture
Mixture of Experts (MoE)
License
MIT
Release Date
1 Nov 2025
Knowledge Cutoff
Oct 2024
API Pricing (per 1M)
Self-hosted only
System Requirements
VRAM requirements for different quantization methods and context sizes
1,024 tokens
Consumer
5x RTX 4090
24GB VRAM
Datacenter
2x NVIDIA A100
80GB VRAM
Apple Silicon
1x Apple M3 Max
128GB VRAM
1,048,576 tokens
Consumer
6x RTX 4090
24GB VRAM
Datacenter
2x NVIDIA A100
80GB VRAM
Apple Silicon
1x Apple M3 Max
128GB VRAM
Architecture Diagram
Evaluation Benchmarks
Rank
#185
| Benchmark | Score | Rank |
|---|---|---|
Intelligence Index | 0.07 | 247 |
Rankings
Overall Rank
#185
Coding Rank
-
About Kimi Linear 48B A3B Instruct
Kimi Linear 48B A3B is an open-source hybrid linear attention MoE model by Moonshot AI activating 3B parameters per token. Powered by Kimi Delta Attention, it achieves high decoding throughput and a 75% reduction in KV cache across a 1M context window.
Technical Specifications
Attention
Attention Structure
Multi-Head Attention
Attention Heads
32
Key-Value Heads
1
Attention Head Dimension
72
Position Embedding
Absolute Position Embedding
RoPE Theta
10,000
Sliding Window Attention
No
Sliding Window Size
-
Sliding Window Ratio
-
Linear Attention
-
Linear Attention Ratio
-
Normalization
RMS Normalization
Activation Function
SwigLU
Dimensions
Auxiliary Parameters
-
Hidden Dimension Size
4,096
Number of Layers
36
FFN Intermediate Size (Dense)
1,024
Multi-Token Prediction Heads
0
Tokenizer
Vocabulary Size
163,840
Mixture of Experts
Total Expert Parameters
3.0B
Number of Experts
128
Active Experts
8
Shared Experts
1
FFN Intermediate Size (per Expert)
1,024
Dense Layers Before MoE
1
About Kimi Linear
Moonshot AI's hybrid linear attention architecture with Kimi Delta Attention for efficient long-context processing.
Other Kimi Linear Models
- No related models available