Phi-3-mini

The Phi-3-mini architecture is explicitly documented in the official technical report as a dense decoder-only Transformer. It utilizes a hidden dimension of 3072, 32 layers, and 32 attention heads. Notably, it adopts the Llama-2 block structure to ensure compatibility with existing community tools. The report details the use of Grouped-Query Attention (GQA) with 4 queries sharing 1 key, and the 128K variant's use of LongRope for context extension is well-documented. While the high-level methodology is clear, specific hyperparameter tuning details for the pretraining phase are less exhaustive than for the 7B variant.

Microsoft discloses that the model was trained on 3.3 trillion tokens (later updated to 4.9T for some variants) consisting of 'textbook-quality' synthetic data and heavily filtered web data. However, the specific proportions of these sources are not provided. The filtering criteria ('educational level') are described conceptually but the exact implementation, specific web domains, or the identity of the models used to generate the synthetic data remain proprietary. No sample data or detailed breakdown of the 3.3T tokens is publicly available.

The model uses the Llama-2 tokenizer with a vocabulary size of 32,064 tokens, which is publicly accessible and well-documented. The technical report explicitly mentions the removal of BoS tokens and the addition of specific chat template tokens (<|system|>, <|user|>, <|end|>, <|assistant|>). The tokenizer is available via the Hugging Face transformers library, allowing for full inspection and verification of tokenization behavior.

The parameter count is precisely stated as 3.8 billion. As a dense model, all parameters are active during inference, and this is clearly communicated. The architectural breakdown (layers, heads, hidden dimensions) is fully disclosed in the technical report, providing a clear map of parameter distribution across the model's components.

Microsoft provides specific hardware and duration details: the model was trained on 512 NVIDIA H100-80G GPUs over a period of 10 days. This allows for a reasonable estimation of total compute (approx. 122,880 GPU hours). However, the report lacks a formal carbon footprint calculation or a detailed breakdown of energy consumption, which are required for a perfect score in this category.

While the technical report lists scores across numerous standard benchmarks (MMLU, GSM8K, HumanEval), it admits that the evaluations were conducted using a 'Microsoft internal tool.' The exact prompts and few-shot examples are not fully disclosed, and the evaluation code is not public. Independent researchers have noted significant performance discrepancies when using standard evaluation harnesses like LM Eval Harness compared to the reported figures.

The model consistently identifies itself as a Microsoft-developed AI and is transparent about its versioning (e.g., distinguishing between 4K and 128K variants). It generally acknowledges its limitations as a small language model, particularly regarding its knowledge cutoff (October 2023) and its primary optimization for English and reasoning tasks.

The model weights and associated code are released under the highly permissive MIT License. This is explicitly stated on the official Hugging Face repository and in the technical report. There are no conflicting commercial restrictions or 'open-ish' terms; it is a standard, legally clear open-source license that allows for broad commercial and research use.

Hardware requirements are well-documented for various deployment scenarios. Microsoft and third-party partners (like ONNX Runtime) provide VRAM estimates for FP16 (~8GB) and 4-bit quantization (~1.8GB to 2GB). The impact of quantization (AWQ vs RTN) on accuracy is discussed in technical blogs, and the model's ability to run on specific mobile hardware (iPhone 14) is verified with performance metrics (12 tokens/sec).

Microsoft has released multiple versions (initial April release, June update, and the subsequent Phi-3.5 series), but they do not consistently use strict semantic versioning. While 'Release Notes' are provided on Hugging Face, they often lack a detailed changelog of specific weight adjustments or data recipe changes. Users have reported 'silent' updates where model behavior changed without a corresponding version increment in the metadata.