Gemma 2 27B

The model architecture is extensively documented in the official technical report ('Gemma 2: Improving Open Language Models at a Practical Size'). It details a decoder-only Transformer with specific innovations: Grouped-Query Attention (GQA), interleaved local sliding window (4096 tokens) and global attention (8192 tokens), and logit soft-capping (capped at 50.0 for attention and 30.0 for final logits). Unlike the 2B and 9B variants which used distillation, the 27B model is explicitly stated to be trained from scratch. The report provides a clear table of hyperparameters including layer counts (46), head counts, and embedding dimensions.

Transparency regarding training data is minimal. Google discloses the total token count (13 trillion) and general categories (web documents, code, science articles), but provides no specific percentage breakdown, source names, or sampling proportions. The methodology for 'data mixture' is vaguely attributed to 'ablations similar to Gemini 1.0' without further detail. While filtering for CSAM and PII is mentioned, the lack of source-level transparency or a dataset card makes the claims unverifiable.

The tokenizer is publicly available on Hugging Face and documented as a SentencePiece-based subword tokenizer with a large vocabulary of 256,128 tokens. It uses byte-level encoding, digit splitting, and preserves whitespace. The vocabulary size and implementation details are consistent across official documentation and third-party libraries (transformers, llama.cpp), allowing for full verification of tokenization behavior.

The model's parameter count is clearly stated as 27B, and it is a dense architecture. Detailed architectural breakdowns are provided in the technical report, specifying the number of layers (46), hidden size (4608), and the specific configuration of GQA heads. There is no ambiguity regarding active vs. total parameters as it is not an MoE model.

Google discloses the hardware used (TPUv5p) and the software stack (JAX and ML Pathways), but fails to provide the total compute budget in terms of TPU-hours or GPU-equivalent hours. No specific carbon footprint calculation or energy consumption data for the 27B training run is provided in the technical report, which only offers high-level '4M' efficiency marketing claims rather than model-specific environmental data.

While the technical report lists scores for standard benchmarks (MMLU, GSM8K, HumanEval), it lacks the specific evaluation code, exact prompts, or few-shot examples required for precise reproduction. Third-party reports have noted significant performance discrepancies when logit soft-capping is not correctly implemented in inference engines, highlighting a lack of detailed reproduction instructions for the model's unique architectural features.

The model consistently identifies as Gemma 2 and maintains a clear versioning identity. It does not exhibit the identity confusion seen in some other open-weight models that claim to be GPT-4. It is transparent about its nature as an AI developed by Google, and its capabilities generally align with its documented performance profile.

The model is released under the 'Gemma Terms of Use,' which is a custom license rather than a standard OSI-approved license like Apache 2.0. While it allows for commercial use and redistribution, it includes specific 'Use Restrictions' and a 'Responsible Use Policy.' The terms are clearly written and publicly accessible, but the custom nature and restrictive clauses prevent a higher score.

Hardware requirements are well-documented by both Google and the community. Official documentation notes that the 27B model can run on a single A100 (80GB) or H100 at full precision. Community documentation on Hugging Face and Ollama provides detailed VRAM requirements for various quantization levels (4-bit, 8-bit) and context lengths, though Google's own documentation focuses primarily on high-end datacenter hardware.

Google uses version numbers (e.g., Gemma 2) and maintains a model card on Hugging Face. However, there is no detailed, public-facing changelog for minor weight updates or iterative 'safety' tuning. Users have reported 'silent' changes in behavior and performance issues related to library compatibility (e.g., Flash Attention vs. Soft-capping) that were not proactively documented in a centralized version history.