Phi-2

Microsoft provides a clear description of Phi-2 as a decoder-only Transformer model with 2.7 billion parameters. The training methodology is detailed in official blog posts and the model card, highlighting a next-word prediction objective and a unique 'scaled knowledge transfer' from its predecessor, Phi-1.5. While the specific architectural modifications (like the use of MixFormer and Flash Attention) are mentioned, a full peer-reviewed technical paper with exhaustive architectural diagrams is absent, though the Hugging Face implementation provides high transparency into the code structure.

The model was trained on 1.4 trillion tokens, and Microsoft discloses the general composition: a mixture of 'textbook-quality' synthetic data (generated by GPT-3.5/4) and filtered web data from sources like Falcon RefinedWeb and SlimPajama. However, the exact percentage breakdown between synthetic and web data is not explicitly provided, and the specific filtering heuristics or the 'textbook' generation prompts remain proprietary, limiting full reproducibility of the dataset.

The tokenizer is publicly accessible via the Hugging Face 'transformers' library. It has a known vocabulary size of 51,200 (with 50,295 active tokens), and its behavior is well-documented in community discussions and official model cards. The use of a standard BPE-based approach is verifiable through the provided 'tokenizer.json' and 'vocab.json' files in the repository.

Phi-2 is a dense model with a clearly stated 2.7 billion parameters. Unlike MoE models, there is no ambiguity regarding active vs. total parameters. The architectural breakdown (layers, heads, embedding dimensions) is fully transparent through the configuration files on Hugging Face, and the impact of its compact size on performance is the central theme of its documentation.

Microsoft discloses that the model was trained for 14 days using 96 NVIDIA A100-80G GPUs. While this provides a clear hardware and duration metric, there is no official disclosure of the total carbon footprint, energy consumption in MWh, or the specific cost of the training run, which are key requirements for high scores in this category.

While Microsoft provides extensive benchmark results (MMLU, GSM8K, HumanEval) and specifies the few-shot settings (e.g., 5-shot for MMLU), the exact evaluation code and full prompt sets used for these internal evaluations are not publicly released in a single reproducible repository. Third-party evaluations on the Open LLM Leaderboard provide some verification, but discrepancies in scoring across different versions of benchmarks are noted.

Phi-2 demonstrates high identity consistency. It is a base model without instruction tuning, yet it does not typically hallucinate being a competitor's model (like GPT-4) in standard completions. It is clearly versioned within the Phi family, and its limitations as a non-aligned base model are explicitly stated in the 'Intended Uses' and 'Limitations' sections of its documentation.

The model is released under the MIT License, which is a highly permissive, standard open-source license allowing for commercial use, modification, and distribution. This was a significant upgrade from its initial restricted research license, and the current terms are clear, public, and unambiguous.

Hardware requirements are well-documented by both Microsoft and the community. VRAM requirements for FP16 (approx. 5.2 GB) and various quantization levels (e.g., 4-bit requiring ~1.8 GB) are widely available. Documentation includes guidance on using Flash Attention to optimize memory and performance, and the model's suitability for consumer-grade GPUs is a verified claim.

Phi-2 follows a clear naming convention within the Phi family (Phi-1 -> 1.5 -> 2). However, it lacks a formal, granular changelog for weight updates or minor iterations. While the Hugging Face repository tracks file changes, there is no structured semantic versioning for the model weights themselves, making it difficult to track subtle 'silent' updates if they occur.