Phi-1

The model's architecture is extensively documented in the 'Textbooks Are All You Need' paper and official model cards. It is a decoder-only Transformer with 1.3 billion parameters, 24 layers, a hidden dimension of 2048, and 32 attention heads. Specific technical choices like Rotary Position Embedding (RoPE) and FlashAttention are explicitly disclosed. The training methodology, including the two-stage process (pretraining on 'CodeTextbook' and finetuning on 'CodeExercises'), is clearly described with step counts and learning rate schedules.

Microsoft provides a detailed breakdown of the training data: 6 billion tokens of filtered web code (from The Stack and StackOverflow), 1 billion tokens of synthetic 'textbook' data generated by GPT-3.5, and 180 million tokens of synthetic exercises. While the exact filtering classifier and the full synthetic dataset are not public, the proportions and sources are disclosed with high specificity compared to industry standards.

Phi-1 uses the same tokenizer as the CodeGen-350M-mono model, which is publicly accessible. The vocabulary size is stated as 51,200 (padded for GPU efficiency from a base of ~50,257). Documentation on the tokenizer's integration within the Hugging Face 'transformers' library is comprehensive, allowing for direct inspection of tokenization behavior and vocabulary mapping.

The model is a dense architecture with a clearly stated 1.3 billion total parameters. Detailed architectural specifications, including the MLP-inner dimension (8192) and attention head dimensions (64), are provided in the technical paper. There is no ambiguity regarding active vs. total parameters as it is not an MoE model.

Training compute is well-documented: the model was trained on 8 Nvidia A100 GPUs. Pretraining took approximately 4 days (770 GPU hours), and finetuning took an additional 7 hours. While specific carbon footprint calculations or total dollar costs are not in the primary paper, the hardware and time metrics allow for reliable third-party estimation.

While the paper reports clear scores on HumanEval (50.6%) and MBPP (55.5%), it lacks the full release of the evaluation code and exact prompt templates used for these specific results. Independent researchers have noted challenges in reproducing these exact figures due to sensitivity to prompt formatting and the lack of a standardized evaluation harness at the time of release. (Score adjusted for known issues).

Phi-1 consistently identifies as a research model specialized for Python. It does not exhibit identity confusion with larger models like GPT-4, despite using GPT-3.5 for synthetic data generation. The model card explicitly defines its scope as a 'text-to-code' model and warns against its use for general conversation or production coding.

The model is released under the MIT License, which is a highly permissive, standard open-source license. This allows for commercial use, modification, and distribution with minimal restrictions. The licensing terms are clear and consistent across the GitHub repository and Hugging Face model card.

VRAM requirements are well-understood due to the model's small size (approx. 2.6GB for weights in FP16). Documentation and community testing provide clear guidance on running the model on consumer hardware (e.g., RTX 3060). However, official documentation on quantization-specific accuracy tradeoffs (e.g., 4-bit vs 8-bit) is less detailed than the architectural specs.

The model follows a basic versioning structure (Phi-1, Phi-1.5, etc.), but lacks a detailed, granular changelog for weight updates or minor revisions. While the initial release is stable, there is limited infrastructure for tracking silent updates or behavioral drift over time within the same version identifier.