Gemma 1 2B

Gemma 1 2B is extensively documented in the official technical report ('Gemma: Open Models Based on Gemini Research and Technology'). The architecture is explicitly defined as a decoder-only transformer with 18 layers, a hidden dimension of 2048, and 8 attention heads. It uniquely utilizes Multi-Query Attention (MQA) for the 2B variant, distinct from the 7B's Multi-Head Attention. Key modifications like RoPE embeddings, GeGLU activations, and RMSNorm are clearly stated and justified. The relationship to the Gemini family is transparent, though the specific 'distillation' or training recipe details from the larger Gemini models are described at a high level rather than with full procedural reproducibility.

While Google discloses the total token count (2 trillion for the 2B model) and general categories (web documents, mathematics, and code), it fails to provide a specific percentage breakdown or name the exact datasets used. The documentation mentions filtering for CSAM, PII, and quality using model-based classifiers, but these methodologies are not public. The lack of specific data sources or a detailed composition breakdown (e.g., 'StackOverflow: 5%') prevents independent verification of the training data's diversity or bias.

The tokenizer is a SentencePiece-based model with a large vocabulary of 256,128 tokens, which is publicly accessible via Hugging Face and the official GitHub repository. Documentation specifies technical details such as digit splitting, byte-level encoding for unknown tokens, and the preservation of whitespace. The vocabulary size and tokenization approach are consistent across official documentation and third-party implementations like Transformers and vLLM.

The model is marketed as '2B', but technical documentation reveals the actual parameter count is approximately 2.5 billion. While the 'active' vs 'total' distinction is not applicable here as it is a dense model, the discrepancy between the marketing name and the actual size is documented in the technical report (Table 1). The architectural breakdown (layers, heads, embedding dimensions) is fully transparent, allowing for precise parameter calculation by researchers.

Google discloses the hardware used (TPUv5e) and the scale (512 TPUv5e chips across 2 pods for the 2B model). However, it does not provide the total training duration in hours, the total energy consumption, or the carbon footprint. While the 'Pathways' approach and sharding techniques are mentioned, the lack of specific compute-time metrics or environmental impact data results in a low score for this pillar.

Google provides a wide array of benchmark results (MMLU, GSM8K, HumanEval, etc.) in the technical report. However, the exact prompts, few-shot examples, and evaluation code were not initially released in a centralized, reproducible format. Third-party audits (e.g., Unsloth) discovered significant discrepancies and bugs in the initial release's implementation of the technical report's specifications (such as BOS token requirements and RoPE precision), which hindered immediate reproducibility. The score is further adjusted due to documented evidence of benchmark contamination in the training data.

Gemma 1 2B demonstrates high identity consistency. It correctly identifies itself as a model developed by Google and is transparent about its versioning (e.g., distinguishing between 1.0 and 1.1). There are no significant reports of the model claiming to be a competitor's product (like GPT-4). It maintains a clear boundary regarding its capabilities as a text-only model compared to the multimodal Gemini models.

The model is released under the 'Gemma Terms of Use,' which is a custom 'open weights' license rather than a standard OSI-approved open-source license like Apache 2.0. While it allows for commercial use and redistribution, it includes restrictive clauses regarding 'Model Derivatives' and a 'Prohibited Use Policy' that Google can enforce remotely. The terms are legally clear but create a 'viral' effect where any model trained on Gemma output must also follow these terms, leading to some community ambiguity.

Hardware requirements are well-documented by both Google and the community. Official model cards provide VRAM estimates for different precisions (e.g., ~4.7GB for BF16), and third-party tools like the Hugging Face Model Memory Utility provide granular data for quantization (e.g., ~1.2GB for INT4). The impact of context length (8k tokens) on memory is also publicly verifiable through standard transformer memory scaling formulas.

Google maintains a release log and uses version numbers (1.0, 1.1). However, the transition from 1.0 to 1.1 involved significant 'silent' changes in behavior due to a new RLHF method and bug fixes that were not fully detailed in a technical changelog. While the previous versions remain accessible, the lack of a detailed, line-by-line changelog for weights and alignment updates makes tracking drift difficult for developers.