Kimi Linear 48B A3B Instruct

The model's architecture is extensively documented in the technical report 'Kimi Linear: An Expressive, Efficient Attention Architecture' (arXiv:2510.26692). It details a hybrid linear attention mechanism (Kimi Delta Attention or KDA) interleaved with Multi-Head Latent Attention (MLA) in a 3:1 ratio. The report provides mathematical formulations for the KDA layers, including the use of Diagonal-Plus-Low-Rank (DPLR) transition matrices and channel-wise gating. The implementation is publicly available via the 'fla-core' library and official GitHub repository, allowing for architectural verification.

While the total token count (5.7 trillion tokens) and the name of the pretraining corpus (K2) are disclosed, specific composition details such as the percentage breakdown of web data, code, and books are missing. There is no public documentation on the specific filtering, cleaning, or deduplication methodologies used for the K2 corpus, nor is there a sample of the training data available for public inspection. The information remains at a high-level 'marketing' description of 'high-quality data'.

The tokenizer is publicly accessible through the Hugging Face repository and can be loaded using standard 'AutoTokenizer' calls. It is based on the same tokenizer used in the Kimi K2 series, with a known vocabulary size and support for multilingual inputs. Documentation exists regarding its integration with the 'fla-core' library, though a detailed report on its training data alignment is not explicitly provided in the main technical paper.

Moonshot AI provides precise details regarding parameter density. The model is a Mixture-of-Experts (MoE) with 48 billion total parameters and approximately 3 billion active parameters per forward pass. The architecture activates 8 out of 256 experts per token, including one shared expert. This level of detail regarding sparse vs. dense parameter counts is exemplary and verifiable through the model's configuration files.

Information regarding training compute is extremely limited. While the technical report mentions the use of the 'Muon' optimizer and 'WSD' learning rate schedule, it does not disclose the total GPU/TPU hours, the specific hardware cluster used (e.g., number of H100s), or the training duration. Furthermore, there is no public calculation of the carbon footprint or estimated financial cost of the training run.

The technical report includes results for several standard benchmarks (MMLU-Pro, RULER, RepoQA, MATH500) and specifies the context lengths used (e.g., 4k for MMLU-Pro, 128k for RULER). However, the exact evaluation prompts and few-shot examples are not fully disclosed in a dedicated evaluation repository. While some third-party verification is available through community testing on platforms like OpenRouter, a fully reproducible evaluation suite is not provided.

The model consistently identifies as 'Kimi' or a Moonshot AI product in standard instruction-following tasks. It maintains version awareness (Kimi Linear) and is transparent about its hybrid architecture and 1-million-token context window capabilities. There are no documented cases of the model claiming to be a competitor's model (e.g., GPT-4) or denying its nature as an AI.

The model weights and the KDA kernel code are released under the MIT License, which is a standard, highly permissive open-source license. The terms are clear, allowing for both commercial use and derivative works without the restrictive 'semi-free' clauses found in previous Moonshot AI releases. This represents the highest level of licensing transparency.

Moonshot AI and community contributors have provided detailed VRAM requirements for various configurations. Official documentation states that the model can run on a single 80GB VRAM GPU (like an A100 or H100) for basic inference. Third-party documentation on Hugging Face (e.g., cyankiwi) provides specific VRAM footprints for 4-bit AWQ quantization (approx. 28.4 GB) vs. full precision (approx. 91.5 GB), including context length scaling estimates.

The model uses a clear naming convention (Kimi-Linear-48B-A3B-Instruct), but there is no public, centralized changelog or semantic versioning system for tracking updates to the weights or the underlying KDA kernels. While the GitHub repository tracks code changes, the 'silent' update of model weights on Hugging Face remains a possibility, and there is no formal deprecation policy for older checkpoints.