Gemma 1 7B

Google provides a comprehensive technical report and official blog posts detailing the Gemma 1 7B architecture. It is explicitly described as a decoder-only transformer with specific modifications: Multi-Head Attention (MHA), Rotary Positional Embeddings (RoPE), GeGLU activation, and RMSNorm. The report details the layer count (28), hidden dimension (3072), and head size (256). While it mentions being built on Gemini research, the specific training methodology (pretraining from scratch on 6T tokens) is clearly stated, though the exact 'Gemini-inspired' architectural nuances remain partially proprietary.

The training data is described as a 6-trillion-token corpus consisting of web documents, code, and mathematics. However, Google provides no specific percentage breakdown of these components (e.g., what portion is code vs. web). While they document high-level filtering techniques for CSAM and sensitive data, the specific sources of the 'web documents' are not named, and no sample data is provided for public inspection. This falls under 'general categories mentioned' without detailed composition.

The tokenizer is publicly available via the official GitHub and Hugging Face repositories. It uses a SentencePiece model with a large vocabulary size of 256,128 tokens. Documentation explicitly states it uses byte-level fallback, splits digits, and preserves whitespace. The vocabulary size and approach are consistently reported across all official sources, and the tokenizer can be directly inspected and tested by the public.

While marketed as '7B', the technical documentation and third-party inspections reveal the model actually has approximately 8.54 billion total parameters (8.5B with weight tying). The '7B' label refers to non-embedding parameters. While the architectural breakdown (layers, heads) is provided, the discrepancy between the marketing name and the actual parameter count is a transparency gap, though the technical report does acknowledge the 8.5B figure in its specifications table.

Google discloses the hardware used (TPUv5e) and the scale (4096 chips across 16 pods for the 7B model). However, they do not disclose the total training duration (hours/days), the total compute cost, or the specific carbon footprint for the Gemma 1 7B training run. Information is limited to hardware type and general infrastructure sharding techniques (ZeRO-3, Pathways).

The technical report lists scores for standard benchmarks (MMLU, GSM8K, HumanEval, etc.), but it does not provide the exact evaluation code, specific prompts, or few-shot examples used to achieve those scores. Third-party researchers have noted significant performance discrepancies and sensitivity to prompt formatting (e.g., BOS token requirements) that were not fully documented in the initial release. Scoring is further reduced due to evidence of benchmark contamination in the training set.

The model consistently identifies itself as Gemma and is version-aware in its instruction-tuned variants. There are no widespread reports of the model claiming to be a competitor (like GPT-4) or denying its AI nature. It maintains a coherent identity across official APIs and local deployments, though its self-knowledge of its own parameter count can sometimes reflect the '7B' marketing label rather than the technical 8.5B count.

Gemma is released under a custom 'Gemma Terms of Use' rather than a standard OSI license like Apache 2.0. While it explicitly allows commercial use and redistribution for organizations of all sizes, it includes 'responsible use' restrictions that create legal ambiguity compared to true open-source licenses. The source code for inference is Apache 2.0, but the weights are governed by the more restrictive custom terms.

Google provides general guidance on hardware (runs on laptops/workstations), and the model card lists the bfloat16 precision. However, detailed VRAM scaling for different context lengths and specific quantization-accuracy tradeoff tables are missing from official documentation. Users must rely on third-party community benchmarks to determine that the 7B model effectively requires ~16GB-35GB VRAM depending on the implementation and quantization.

Google maintains a release page and uses versioning (e.g., Gemma 1.0 vs 1.1). However, the changelogs are relatively high-level (e.g., 'improved safety', 'fixed bug in multi-turn'). There is limited documentation on how specific weight updates affect model drift or performance on specific tasks, and previous versions are not always easily accessible once a new 'point' release (like 1.1) becomes the default.