Gemma 4 26B A4B

Gemma 4 26B A4B is explicitly documented as a Mixture-of-Experts (MoE) model derived from the Gemini 3 research lineage. Technical documentation details a hybrid attention mechanism alternating between local sliding-window (1024 tokens) and global full-context layers. It utilizes 128 experts with a routing policy that activates 8 experts plus 1 shared expert per token. While the high-level architecture is well-described across official blog posts and model cards, a formal peer-reviewed technical paper with full ablation studies is currently absent, preventing a higher score.

Disclosure regarding training data is limited to vague marketing claims. Documentation states the model was trained on a 'diverse' dataset supporting over 140 languages and interleaved multimodal inputs (text and images). However, there is no public breakdown of data sources (e.g., percentages of web, code, or academic data), no detailed filtering/cleaning methodology, and no disclosure of the specific proportions of synthetic vs. organic data used. The lack of a technical report leaves these critical details unverifiable.

The tokenizer is publicly accessible via the official GitHub repository and Hugging Face collections. It supports a large vocabulary consistent with the claimed 140+ language support. Technical specifications for tokenization of multimodal inputs (variable resolution image tokens) are documented, with supported visual token budgets (70, 140, 280, 560, 1120) clearly stated. Integration with standard libraries like Transformers and vLLM allows for independent verification of tokenization behavior.

Google provides exemplary transparency regarding parameter density for this variant. The model is clearly labeled '26B A4B', explicitly denoting 25.2B total parameters with 3.8B active parameters per inference. Documentation further clarifies the expert structure (128 total experts, 8 active per token) and the use of a shared expert. This level of detail prevents the common MoE 'parameter inflation' confusion and provides clear expectations for both memory (total params) and compute (active params).

Information regarding training compute is almost entirely absent. While documentation mentions the model can be fine-tuned on TPUs and H100s, there is no disclosure of the total GPU/TPU hours required for the initial pre-training, no hardware cluster specifications used for the primary run, and no carbon footprint or environmental impact calculations. This represents a significant transparency gap typical of proprietary-derived models.

Official benchmarks (MMLU-Pro: 82.4%, GSM8K: 94.1%) are provided with some methodological notes, such as the use of fixed temperature (0.1) and top-p (0.95) sampling. However, the evaluation code itself is not fully centralized in a reproducible repository, and specific few-shot prompts used for all frontier benchmarks are not exhaustively disclosed. Third-party verification on leaderboards like Open LLM Leaderboard and Arena AI provides some external validation, but the lack of a technical paper limits full reproducibility.

The model demonstrates high identity consistency, correctly identifying itself as a member of the Gemma 4 family and acknowledging its MoE architecture in system-level interactions. It is transparent about its versioning and its relationship to the Gemini research line. There are no documented instances of the model claiming to be a competitor's product or misrepresenting its parameter count in self-identification tasks.

Gemma 4 marks a significant shift to a standard, OSI-approved Apache 2.0 license. This is a major transparency improvement over previous 'Gemma Terms of Use' licenses. The terms are clear, publicly accessible, and allow for unrestricted commercial use, modification, and redistribution without revenue caps or usage restrictions. The license is consistently applied across weights, code, and documentation.

Hardware requirements are extensively documented for various quantization levels (FP16, Q8, Q4). Official and community documentation (e.g., Unsloth, vLLM) provides specific VRAM targets: ~18GB for 4-bit and ~60GB for BF16. Crucially, the documentation includes memory scaling data for the 256K context window, noting that the hybrid attention mechanism allows for more efficient VRAM usage at long contexts compared to standard dense models. Quantization trade-offs (e.g., <2.8% loss at 4-bit) are also disclosed.

The model uses basic versioning (Gemma 4 26B A4B), but a comprehensive, public changelog for weight updates or 'silent' fine-tuning refreshes is not maintained. While the release date is clear, there is no formal infrastructure for tracking behavioral drift over time or accessing specific 'checkpoint' versions beyond the initial release. This makes it difficult for developers to ensure long-term stability in production environments.