Falcon-1B

The Falcon3-1B-Instruct model is explicitly documented as a causal decoder-only Transformer with 18 decoder blocks. TII provides specific technical details including the use of Grouped Query Attention (GQA) with 8 query heads and 4 key-value heads, a head dimension of 256, SwiGLU activation, and RMSNorm. Crucially, the model's provenance is described as being 'pruned and healed' from a larger 3B Falcon model using knowledge distillation, which is a more transparent disclosure of origin than many 'trained from scratch' claims. However, the full pretraining procedure for the parent 3B model is only partially detailed in the general Falcon 3 blog post.

TII discloses that the model was trained on 80 Gigatokens of data for the 'healing' phase and the instruct version was post-trained on 1.2 million samples. General categories are provided (web, code, STEM, multilingual), but specific percentage breakdowns of the 80GT or the 14T tokens used for the base 7B model are not publicly available. The data collection and filtering methodologies are mentioned in marketing terms ('curated', 'high-quality') without technical documentation or public access to the underlying datasets.

The tokenizer is publicly accessible via Hugging Face and the vocabulary size is clearly stated as 131,072 tokens. It is documented as a BPE-based tokenizer with support for English, French, Spanish, and Portuguese. The alignment with the claimed language support is verifiable through the provided configuration files and the 'transformers' library integration, though detailed training data for the tokenizer itself is not explicitly separated from the general pretraining data.

The model is a dense architecture with a clearly stated parameter count of approximately 1 billion. Unlike MoE models, there is no ambiguity between active and total parameters. The architectural breakdown (18 layers, specific head counts) is provided in the model card, allowing for a precise understanding of parameter distribution across the network.

TII discloses the hardware used (256 H100 GPU chips) for the pruning and healing phase of the 1B model. However, the total GPU hours, training duration, and specific energy consumption or carbon footprint for the 1B variant are not provided. While the 7B base model's compute is mentioned (1024 H100s for 14T tokens), the lack of specific metrics for the 1B-Instruct variant's post-training and healing phases leaves significant gaps.

Benchmark results are reported for standard sets like MMLU, ARC, and GSM8K. TII specifies the use of the 'lm-evaluation-harness' and mentions the use of chat templates and few-shot settings. However, the exact prompts, few-shot examples, and specific evaluation code for reproducing the reported 'raw scores' are not fully public, and third-party verification is limited to leaderboard entries rather than independent audits.

The model consistently identifies itself as Falcon3 from TII in its system prompts and documentation. There is clear version tracking within the Falcon 3 family (1B, 3B, 7B, 10B) and a distinction between Base and Instruct variants. The model does not exhibit identity confusion with competitors like Llama or GPT in official documentation or standard testing environments.

The model is released under the 'TII Falcon-LLM License 2.0'. While based on Apache 2.0, it includes a custom 'Acceptable Use Policy' and specific terms that are not standard open-source. The license allows for commercial use but includes restrictions and requirements (such as attribution and compliance with the AUP) that make it a 'weights-available' license rather than a pure open-source license, leading to some ambiguity for enterprise users.

VRAM requirements are well-documented by the community and partially by TII through the release of GGUF, AWQ, and GPTQ variants. The model card specifies the 8K context length for the 1B variant, and memory scaling for this context is predictable. However, official documentation lacks a comprehensive table of VRAM vs. batch size vs. quantization levels, relying instead on third-party implementations like Ollama and llama.cpp for this data.

The model has a clear release date (December 2024) and is part of a numbered family. However, there is no public changelog or semantic versioning for the weights themselves (e.g., v1.0 vs v1.1). While the initial release is well-documented, the infrastructure for tracking future silent updates or performance drift is not explicitly presented to the public.