SmolLM3 3B

The model architecture is extensively documented as a decoder-only Transformer based on the Llama design with specific, well-defined modifications. These include Grouped Query Attention (GQA) with 4 key-value heads and a unique 'No Positional Encoding' (NoPE) approach applied in a 3:1 layer ratio. Technical specifications such as 36 hidden layers, a hidden dimension of 2048, and tied embeddings are publicly available. The training methodology is described as a three-stage curriculum (Stable, Mid-training, and Post-training) with clear objectives for each phase.

Hugging Face has provided a high level of transparency regarding the 11.2 trillion token training corpus. The data mixture is broken down by stage (e.g., Stage 1: 85% web, 12% multilingual) and specific public datasets are named, including FineWeb-Edu, DCLM, FineWeb2, and The Stack. The transition between stages and the inclusion of specific reasoning datasets like OpenMathReasoning and synthetic data from Qwen3-32B are documented. While the exact per-file filtering code isn't fully public, the methodology and ratios are exemplary for the industry.

The tokenizer is fully accessible via the Hugging Face library and the 'smollm' GitHub repository. It features a vocabulary size of 49,152 tokens and was trained specifically on the SmolLM corpus to ensure alignment with the training data. Documentation confirms support for six primary languages (English, French, Spanish, German, Italian, Portuguese) and includes the chat template logic for dual-mode reasoning, which is verifiable through the public 'tokenizer_config.json'.

The model is a dense architecture with a clearly stated 3.0 billion parameters. Detailed architectural breakdowns are provided, including the hidden size (2048), intermediate size (11008), and the specific configuration of attention heads (16 query, 4 KV). Because it is a dense model, there is no ambiguity regarding active vs. total parameters, and the impact of architectural choices like tied embeddings on the parameter count is explicitly mentioned.

Hugging Face disclosed the specific hardware used (384 H100 GPUs) and the training duration (24 days), totaling approximately 220,000 GPU hours. The training framework (nanotron) and data processing tools (datatrove) are also public. While a specific carbon footprint calculation or exact dollar cost was not provided in the primary model card, the hardware and time metrics allow for high-fidelity third-party estimation.

The model release includes results for a wide range of standard benchmarks (HellaSwag, ARC, MMLU-Pro, etc.) and specifies the use of the 'lighteval' framework for evaluation. However, while the evaluation datasets are listed in a public collection, the exact prompt versions and few-shot configurations for every single reported score are not consolidated in a single 'reproducibility' file, requiring some effort to reconstruct from the lighteval configurations.

SmolLM3-3B exhibits high identity consistency, correctly identifying its version and origin in official documentation and through its specialized chat template. The model is transparent about its dual-mode reasoning capabilities (think/no-think) and its limitations as a 3B parameter model. There are no documented instances of the model claiming to be a competitor's product or misrepresenting its scale.

The model is released under the Apache 2.0 license, which is a standard, highly permissive open-source license. There are no conflicting terms, and commercial use, modification, and distribution are explicitly permitted. The license applies clearly to both the model weights and the associated code in the GitHub repository.

Official documentation provides clear guidance on hardware requirements, noting that the model can run on devices with as little as 4GB-8GB of RAM. VRAM usage for inference is well-understood given the 3B parameter count (~6GB in FP16), and the model card explicitly mentions support for quantization (4-bit/8-bit) via bitsandbytes and llama.cpp, with community-verified benchmarks for these formats.

The model follows a clear versioning lineage (SmolLM -> SmolLM2 -> SmolLM3). Changes between versions, such as the increase in context length and the addition of reasoning modes, are well-documented in blog posts and commit histories. While it lacks a formal 'semantic versioning' changelog for minor weight updates, the major architectural and data shifts are transparently communicated.