Phi-3-small

Microsoft provides a detailed technical report (arXiv:2404.14219) specifying that Phi-3-small is a dense decoder-only Transformer with 32 layers and a hidden size of 4096. It explicitly documents the use of Grouped Query Attention (GQA) with 4 queries per key and a unique alternating pattern of dense and blocksparse attention layers to optimize the KV cache. While the base architecture is well-documented, the specific 'blocksparse' implementation details are described at a high level without full source code for the custom kernels used in training.

The model was trained on 4.8 trillion tokens. Documentation mentions three main categories: 1) filtered public web data, 2) high-quality educational/code data, and 3) synthetic 'textbook-like' data. However, Microsoft does not provide a specific percentage breakdown of these sources (beyond a mention of 10% multilingual data) or name the specific datasets used. The 'synthetic data' generation process is described conceptually but lacks the transparency required for verification or reproduction of the data mix.

Phi-3-small uses the tiktoken-based tokenizer with a vocabulary size of 100,352, which is a significant departure from the Llama-based tokenizer used in Phi-3-mini. The tokenizer is publicly accessible via Hugging Face and the vocabulary size is clearly stated in the technical report. It is well-documented as being optimized for multilingual support, though detailed alignment between the tokenizer's training data and the model's 4.8T token corpus is not fully disclosed.

The model is clearly identified as a 7B parameter dense model. Microsoft provides a structural breakdown (32 layers, 32 heads, 4096 hidden dimension). While it uses blocksparse attention, it is not a Mixture-of-Experts (MoE) model, so the distinction between total and active parameters is not applicable here. The documentation is clear, though it lacks a precise parameter count beyond the '7B' marketing label (e.g., 7.39B).

Microsoft discloses the hardware used (1024 H100-80G GPUs) and the training duration (18 days). This allows for a rough estimate of compute resources. However, it fails to provide a calculated carbon footprint or the specific energy efficiency metrics of the cluster. The information is better than most proprietary models but lacks the environmental transparency seen in exemplary open-science projects.

While Microsoft reports scores on standard benchmarks (MMLU, GSM8K, etc.) in the technical report, they explicitly state that the prompts and few-shot examples are part of an 'internal tool' and are not fully public. This significantly hinders third-party reproduction. Furthermore, independent research has highlighted significant performance gaps when using different evaluation pipelines, suggesting the reported numbers are highly sensitive to the undisclosed internal settings.

The model consistently identifies itself as a Microsoft Phi-3 model in system prompts and documentation. It maintains a clear versioning identity within the Phi-3 family (Small vs. Mini vs. Medium). There are no documented cases of the model claiming to be a competitor's product (like GPT-4) or denying its nature as an AI developed by Microsoft.

Phi-3-small is released under the highly permissive MIT License, which is clearly stated on the official Hugging Face repository and Microsoft's blog. The license allows for commercial use, modification, and distribution with minimal restrictions. There are no conflicting 'non-commercial' clauses in the primary license text for the weights.

Microsoft and partners (like NVIDIA) provide VRAM requirements for various deployment scenarios. Documentation exists for FP16 and INT4 (via ONNX/DirectML) requirements. The impact of LongRope for 128K context on memory is discussed, though detailed scaling tables for VRAM vs. context length are primarily provided by third-party community benchmarks rather than a single comprehensive official source.

Microsoft uses a naming convention (e.g., Phi-3-small-8k-instruct) but lacks a strict semantic versioning system for weight updates. While they released a 'June 2024 Update' with a changelog on Hugging Face, updates are often delivered as new model cards rather than tracked versions of a single artifact. This makes it difficult for developers to track silent changes or roll back to specific sub-versions without manual commit tracking.