Gemma 3 270M

Gemma 3 270M is explicitly documented as a dense transformer-based model with 12 layers and a hidden dimension of 1024. The architecture is detailed in the Gemma 3 Technical Report, which describes the use of Rotary Positional Embeddings (RoPE), RMSNorm, and a specific interleaved attention mechanism (ratio of local sliding window to global self-attention). It is clearly identified as a text-only variant within a larger multimodal family, and its relationship to the Gemini research lineage is publicly stated.

Google discloses that the model was trained on 6 trillion tokens of text data, including web documents, code, and mathematics, with a knowledge cutoff of August 2024. While it mentions support for 140+ languages and general data categories, it lacks a specific percentage breakdown of the dataset composition (e.g., exact ratios of code vs. web data). The filtering and cleaning methodologies are described in general terms rather than with reproducible specificity, and the raw training data is not public.

The model uses the same tokenizer as the Gemini family, featuring a large vocabulary of 256,000 tokens. This tokenizer is publicly accessible via the 'gemma_pytorch' GitHub repository and Hugging Face. The vocabulary size and its optimization for multilingual support (140+ languages) are well-documented, and the tokenizer's behavior is verifiable through standard library integrations like Hugging Face Transformers and SentencePiece.

The parameter count is precisely disclosed and broken down: 270 million total parameters, with 170 million dedicated to embeddings (due to the large vocabulary) and 100 million for the transformer blocks. As a dense model, all parameters are active during inference, and this distinction is clear in technical communications. The impact of quantization (specifically INT4 via QAT) on model size and memory is also well-documented.

The technical report confirms the use of TPUv4p, TPUv5p, and TPUv5e hardware for training. However, it does not disclose the specific number of GPU/TPU hours consumed, the total energy usage, or the carbon footprint associated with training the 270M variant. While the hardware type is known, the scale of the compute resources remains a high-level estimate rather than a detailed disclosure.

While Google provides benchmark results for HellaSwag, PIQA, and IFEval (51.2%), independent researchers have reported significant difficulties in reproducing these scores, specifically noting a large gap in IFEval performance (20-27% vs the claimed 51.2%). The evaluation code is partially available through the 'lm-evaluation-harness' integration, but the exact prompts and environment configurations required to match official results are not sufficiently detailed to ensure third-party verification.

The model consistently identifies as part of the Gemma 3 family and is transparent about its limitations as a text-only, 270M parameter model. It does not claim the capabilities of its larger multimodal siblings (4B+) and maintains a clear versioning identity. Documentation explicitly warns users about its lack of vision support and its focus on task-specific fine-tuning rather than general-purpose knowledge.

The model is distributed under the 'Gemma Terms of Use,' a custom permissive license that allows for commercial use and redistribution but includes specific prohibited use policies and a 'not open source' attribution due to its restrictive nature compared to standard OSI licenses like Apache 2.0. While the terms are publicly accessible, there is some community confusion caused by the mix of Apache 2.0 for code and custom terms for weights.

Hardware requirements are exceptionally well-documented. Google provides specific VRAM estimates for FP16 (540MB) and INT4 (125MB-200MB) precisions. Battery impact on mobile devices (0.75% for 25 conversations on Pixel 9 Pro) and compatibility with low-resource hardware like Raspberry Pi are explicitly stated and verified by third-party deployment tools like Ollama and LM Studio.

The model follows a basic versioning structure (Gemma 3 270M), and weights are hosted on Hugging Face with commit histories. However, there is no detailed public changelog or semantic versioning system that tracks specific architectural or weight adjustments over time. Documentation for 'drift' or performance changes between the initial release and subsequent checkpoints is minimal.