趋近智
Gemma 4 12B
参数
11.95B
上下文长度
262.144K
模态
Multimodal
架构
Dense
许可证
Apache-2.0
发布日期
3 Jun 2026
训练数据截止日期
-
技术规格
注意力
注意力结构
Multi-Head Attention
注意力头
16
键值头
8
注意力头维度
256
位置嵌入
Absolute Position Embedding
RoPE Theta
10,000
滑动窗口注意力
Yes
滑动窗口大小
1,024
归一化
RMS Normalization
激活函数
GELU
维度
隐藏维度大小
3,840
层数
48
FFN 中间层大小（稠密层）
15,360
多 Token 预测头数
-
分词器
词汇量大小
262,144
架构图
Gemma 4 12B
Google DeepMind's 12B dense open-weights model released June 3, 2026, bridging the gap between the edge-friendly E4B and the more advanced 26B MoE. Uniquely features an encoder-free unified architecture that projects raw image patches and audio waveforms directly into the LLM embedding space through lightweight linear layers, eliminating the latency and memory overhead of separate encoders. Supports 256K token context, native text/image/audio inputs, configurable thinking mode, and runs on consumer laptops with 16GB of RAM.
关于 Gemma 4
Gemma 4 is Google DeepMind's most advanced open model family, built from Gemini 3 research and technology. Featuring both Dense and Mixture-of-Experts (MoE) architectures, these multimodal models handle text, images, and audio (on smaller variants), with context windows up to 256K tokens. Designed for frontier-level performance across reasoning, coding, and agentic workflows, Gemma 4 delivers unprecedented intelligence-per-parameter from mobile devices to enterprise servers. Released under Apache 2.0 license.
其他 Gemma 4 模型
评估基准
没有可用的 Gemma 4 12B 评估基准。