Gemma 3 12B

Google provides a detailed technical report (arXiv:2503.19786) and model cards that explicitly describe the 12B variant's architecture. It is a decoder-only transformer using Grouped-Query Attention (GQA) and a specific 5:1 interleaving of local sliding window and global attention layers to optimize KV-cache. The multimodal integration uses a 400M SigLIP vision encoder with a 'Pan & Scan' strategy for flexible resolutions. The report details the use of Rotary Position Embeddings (RoPE) with frequency scaling (1M for global, 10k for local) and QK-norm instead of soft-capping. While the high-level methodology is clear, the exact layer-by-layer configuration and specific distillation teacher details remain partially proprietary.

The model was trained on 12 trillion tokens of text and image data. Documentation identifies broad categories: web documents (140+ languages), code, mathematics, and image-text pairs. It mentions rigorous filtering for CSAM and sensitive personal information. However, it lacks a granular percentage breakdown of the mixture (e.g., exact ratio of code vs. web) and does not provide specific names or versions of the datasets used, citing general 'diverse internet data' and 'internal datasets' for vision tuning.

The tokenizer is a SentencePiece model with a 262,144 vocabulary size, shared with the Gemini 2.0 series. It is publicly available via the Hugging Face 'transformers' library and Google's official repositories. Documentation specifies technical details such as split digits, preserved whitespace, and byte-level encodings. Its performance across 140+ languages is documented, and the vocabulary is verified to be more balanced for non-English text compared to previous versions.

The model is explicitly defined as a 12.2 billion parameter dense model. The technical report clarifies that the vision encoder is a 400M parameter SigLIP variant, which is shared across the 4B, 12B, and 27B models. This distinction between the language backbone and the vision component provides high transparency regarding active parameter counts during multimodal tasks.

Google discloses the hardware used (TPUv4p, TPUv5p, and TPUv5e) but fails to provide the total number of GPU/TPU hours or the specific duration of the training run. There is no calculated carbon footprint or energy consumption data provided in the technical report or model card, which are key requirements for high scores in this category.

The technical report includes results for standard benchmarks like MMLU, MATH, HumanEval, and MMMU. It specifies shot counts (e.g., 4-shot for MATH) and compares results against previous Gemma versions and competitors. However, the exact evaluation code and the specific prompts/few-shot examples used for all benchmarks are not fully public, making exact third-party reproduction difficult without significant effort.

Gemma 3 12B demonstrates high identity consistency. It is trained to identify as 'Gemma' and correctly acknowledges its developer (Google). It does not claim to be a competitor's model (like GPT-4) and is transparent about its multimodal nature and versioning in its system prompts and documentation.

The model is released under the 'Gemma Terms of Use,' which is a custom open-weights license. It allows for commercial use and redistribution but imposes specific restrictions (e.g., attribution requirements, prohibited use policy, and non-sublicensable terms). While the terms are legally clear, they are more restrictive than standard OSI-approved licenses like Apache 2.0, leading to documented compatibility issues with other open-source software.

Google provides comprehensive VRAM estimates for various precisions (BF16, INT8, INT4) and specific hardware recommendations (e.g., 27.6 GB for BF16, 6.6 GB for INT4). Documentation also covers the memory impact of the 128K context window and the KV-cache optimization. Quantization-Aware Training (QAT) checkpoints are provided with documented accuracy-efficiency trade-offs.

The model uses a clear naming convention (Gemma 3 12B) and distinguishes between base (PT) and instruction-tuned (IT) variants. However, Google does not maintain a public, detailed changelog for minor weight updates or provide a formal policy regarding model drift or deprecation schedules for specific checkpoints.