Phi-4-Mini

The model is explicitly documented as a dense decoder-only Transformer with 3.8 billion parameters. Microsoft provides a technical report detailing specific architectural choices, including the use of 32 Transformer layers, a hidden state size of 3,072, and Grouped-Query Attention (GQA) with 24 query heads and 8 key/value heads. It also documents the use of LongRoPE for context extension and tied input/output embeddings. The training methodology, including supervised fine-tuning (SFT) and Direct Preference Optimization (DPO), is clearly stated in the official documentation.

Microsoft discloses that the model was trained on 5 trillion tokens from a mix of filtered public web data and synthetic data. While they provide general categories (educational data, code, synthetic 'textbook-like' data) and mention that synthetic data is a primary focus for reasoning, they do not provide a precise percentage breakdown of the 5T tokens or specific source names for the 'acquired academic books.' The methodology for data filtering and decontamination is described at a high level, but the exact datasets remain proprietary.

The model uses the 'o200k_base' tiktoken tokenizer with a clearly stated vocabulary size of 200,064 tokens. The tokenizer is publicly available on Hugging Face, and its support for 24 languages is documented and verifiable through the provided configuration files. The transition from the Phi-3.5 tokenizer to this larger vocabulary for better multilingual support is explicitly justified in technical communications.

The model is clearly identified as a dense architecture with 3.8B total parameters. There is no ambiguity regarding active vs. total parameters as seen in MoE models. Detailed architectural specifications, such as the number of layers and head configurations, are provided in the technical report and model cards, allowing for a complete understanding of parameter distribution.

Microsoft provides specific hardware and duration details for the training process: 512 A100-80G GPUs for 21 days. This allows for a reasonable estimation of total compute (approx. 258,000 GPU hours). While they do not provide a direct carbon footprint calculation or exact dollar cost, the disclosure of hardware type, count, and duration is significantly more transparent than most industry peers.

The model is evaluated using OpenAI's SimpleEval framework, which is a public and reproducible standard. Microsoft specifies the versions and settings (e.g., 0-shot, 5-shot, CoT) for major benchmarks like MMLU, GSM8K, and MATH. However, they also reference 'internal benchmarks' for certain capabilities, and the full evaluation code/prompts for all reported metrics are not consolidated in a single public repository for one-click reproduction.

The model generally identifies itself correctly as a Microsoft-developed AI. Technical documentation acknowledges that earlier versions had minor identity confusion issues (claiming to be from other companies) and states that ad-hoc training data was used to correct this in the Phi-4 release. It provides clear versioning (Phi-4-Mini-Instruct) and is transparent about its limitations regarding factual knowledge due to its small size.

The model weights and associated code are released under the highly permissive MIT License. This is explicitly stated on Hugging Face, Azure, and in official blog posts. There are no conflicting commercial restrictions or 'open-ish' custom licenses; it is a standard, legally clear open-source license that allows for commercial use and derivative works without ambiguity.

VRAM requirements are well-documented across multiple sources, including official Microsoft blogs and third-party implementation guides. Requirements for FP16 (approx. 9GB) and various quantization levels (e.g., 4-bit GGUF at ~2.5GB) are publicly available. The impact of the 128K context window on KV cache memory is also addressed through the documentation of GQA and its efficiency gains.

The model uses basic versioning (v1.0) and maintains a release date (February 2025). While there is a 'Release Notes' section on the model card, it lacks a detailed, granular changelog for minor weight updates or specific data mixture adjustments. There is no formal system for tracking performance drift over time, although the model is described as a 'static' release.