Falcon3-1B

The Falcon3-1B architecture is comprehensively documented as a causal decoder-only Transformer with 18 layers. TII explicitly details the use of Grouped Query Attention (GQA) with 8 query heads and 4 key-value heads, SwiGLU activation, RMSNorm, and Rotary Position Embeddings (RoPE). The model's provenance is clearly linked to a 'pruning and healing' methodology derived from larger Falcon 3 models (3B and 7B), which is a significant level of methodological disclosure for a 1B variant.

TII provides a high-level breakdown of the training data, stating it was 'healed' on 80 billion tokens consisting of web, code, STEM, and multilingual data. While it mentions the RefinedWeb dataset as a primary source for the family, the specific proportions for the 1B variant's 80GT healing set are not disclosed. The instruct version mentions 1.2 million samples for post-training, but detailed source lists or data filtering scripts for this specific variant are absent.

The tokenizer is publicly available via Hugging Face and is fully integrated into the 'transformers' library. It features a large vocabulary of 131,072 tokens, supporting English, French, Spanish, and Portuguese. Technical details such as the use of Byte Pair Encoding (BPE) and specific special tokens are well-documented in the model card and configuration files.

The model is a dense architecture with approximately 1B parameters (specifically cited as 1.67B total in some technical manifests like Ollama, though marketed as 1B). TII provides a detailed architectural breakdown including layer counts (18), head dimensions (256), and attention configurations, which allows for precise verification of parameter distribution across the model.

TII discloses that the 1B model was pruned and healed using 256 H100 GPU chips. However, the total training duration (GPU hours) and the associated carbon footprint or energy consumption for this specific variant are not explicitly provided. While hardware types are named, the lack of duration or environmental impact metrics prevents a higher score.

While TII provides a table of internal benchmark results (MMLU, GSM8K, etc.) and mentions using the 'lm-evaluation-harness', they do not release the specific evaluation code, exact prompts, or few-shot examples used to achieve these scores. This limits third-party ability to replicate the exact reported figures. (Score adjusted for discovered external research indicating potential contamination risks in the model family).

The model consistently identifies as part of the Falcon 3 family from TII in its system prompts and documentation. It maintains a clear versioning distinction between 'Base' and 'Instruct' variants. There is no evidence of the model claiming to be a competitor's product or misrepresenting its fundamental nature as an AI.

The model is released under the 'TII Falcon-LLM License 2.0'. While the license is publicly accessible and based on Apache 2.0, it includes custom clauses and an Acceptable Use Policy. There is significant community debate regarding its 'open source' status due to commercial restrictions (royalty obligations for high-revenue entities), which creates ambiguity compared to standard OSI-approved licenses.

Hardware requirements are exceptionally well-documented. TII and community partners provide specific VRAM requirements for various quantization levels (FP16, INT8, INT4) and context lengths. The model's efficiency on consumer hardware like laptops is a primary focus of its documentation, with clear guidance on deployment via tools like llama.cpp and Ollama.

The model uses a clear naming convention (Falcon3-1B-Instruct), but a formal, public changelog tracking silent updates or weight drifts is not readily available. While the release date is clear (December 2024), there is no established infrastructure for users to track ongoing updates or access specific historical snapshots beyond the initial release.