Kimi-VL-A3B-Thinking

The model's architecture is extensively documented in the Kimi-VL Technical Report (arXiv:2504.07491). It explicitly names the base language model as 'Moonlight' (a 16B MoE model) and the vision encoder as 'MoonViT' (based on SigLIP-SO-400M). The report details the integration via a two-layer MLP projector and the use of 2D Rotary Positional Embeddings (RoPE) to handle native-resolution images. The training methodology is broken down into four distinct stages: standalone ViT training, joint pre-training (1.4T tokens), joint cooldown, and long-context activation.

The technical report provides a high-level overview of the training data, noting a total of 4.4T tokens used during the joint training stages. It mentions the use of pure text data, image-text pairs, and interleaved multimodal data. While it specifies that 'long data' was upsampled to 25% during the long-context activation stage, it lacks a granular percentage breakdown of specific sources (e.g., exact ratios of web, code, or specific academic datasets) and does not provide a comprehensive list of all data sources used, citing 'rigorous individual validation' instead of full disclosure.

The tokenizer is publicly accessible via the Hugging Face repository and the official GitHub. It is based on the DeepSeek-V3/V2 tokenizer architecture, which is well-documented. The vocabulary size and tokenization approach (byte-level BPE) are verifiable through the provided `config.json` and `tokenizer_config.json` files. The model's support for multilingual inputs (English, Chinese, etc.) is consistent with the tokenizer's design and is verified in technical documentation.

Moonshot AI is highly transparent about the model's MoE structure. It clearly states a total parameter count of 16 billion with approximately 2.8 billion active parameters per token. The documentation further specifies the expert configuration (64 routed experts, with 8 selected per token) and the use of Multi-head Latent Attention (MLA), which is a level of detail rarely seen in model releases. This prevents the common 'parameter inflation' seen in other MoE models.

While the technical report mentions the use of a '4D parallelism strategy' (Data, Expert, Pipeline, and Context Parallelism) and the 'Muon' optimizer, it conspicuously lacks specific details on the total compute budget. There is no disclosure of the number of GPU/TPU hours, the specific hardware cluster size used for the full 4.4T token training, or the estimated carbon footprint. Information is limited to general technical optimizations rather than resource consumption.

The model provides comprehensive benchmark results across 24+ datasets (MMMU, MathVision, OSWorld, etc.) in the technical report. Crucially, Moonshot AI published 'Best Practices for Benchmarking' which includes recommended settings (temperature, top_p, max tokens) and specific 'thinking' configurations to ensure reproducible results. Evaluation code and example inference scripts are available on GitHub, though some third-party verification is still pending for the newest '2506' variant.

The model exhibits strong identity consistency, correctly identifying itself as a Kimi-series model developed by Moonshot AI. It distinguishes between its 'Instruct' and 'Thinking' variants and is aware of its versioning (e.g., the 2504 vs 2506 updates). There are no documented cases of the model claiming to be a competitor's architecture (like GPT-4) or denying its nature as an AI.

The model, weights, and source code are all explicitly released under the MIT License, as stated on the official GitHub, Hugging Face model card, and technical report. This is a highly permissive, standard open-source license with no 'open-weights-only' restrictions or conflicting commercial terms, providing maximum transparency and freedom for derivative works.

Hardware requirements are well-documented by both the provider and the community. Official documentation notes that 50GB VRAM is needed for single-GPU LoRA fine-tuning. Community resources (Ollama/Replicate) provide specific VRAM estimates for different quantization levels (e.g., ~16GB for Q4_K_M, ~40GB for FP16). The impact of the 128K context window on memory is also acknowledged, though a formal scaling chart from the provider is missing.

Moonshot AI uses a clear date-based semantic versioning system (e.g., Kimi-VL-A3B-Thinking-2506). They maintain a detailed changelog in their technical blogs and GitHub 'News' section, documenting specific improvements such as the 20% reduction in thinking length and the 4x increase in supported image resolution. Previous versions remain accessible on Hugging Face, allowing users to track and mitigate potential performance drift.