Falcon3-3B

The model is explicitly identified as a transformer-based causal decoder-only architecture. TII provides specific details on its derivation, noting it was pruned and 'healed' from the larger Falcon3-7B-Base model using knowledge distillation. Architectural specifics are well-documented, including the use of 22 decoder blocks, Grouped Query Attention (GQA) with 12 query and 4 KV heads, SwiGLU activation, RMSNorm, and Rotary Position Embeddings (RoPE) with a specific high value (1000042) for context handling.

While TII discloses the total token count for the Falcon 3 family (14 trillion) and the specific amount used for 'healing' the 3B variant (100 billion tokens), the breakdown of the dataset is only described in general categories: web, code, STEM, and high-quality multilingual data. Specific proportions, source names beyond the legacy 'RefinedWeb', and detailed filtering/cleaning methodologies for this specific version are not publicly detailed in a technical paper.

The tokenizer is publicly accessible via the Hugging Face repository and is well-documented. It features a vocabulary size of 131,072 tokens, which is a significant increase from previous versions. The approach is consistent with the claimed support for English, French, Spanish, and Portuguese, and the tokenizer files (tokenizer.json, tokenizer_config.json) are available for direct inspection and verification.

The model's parameter count is clearly stated as 3 billion. As a dense model, all parameters are active during inference. TII provides a detailed architectural breakdown including the number of layers (22), head dimensions (256), and attention configurations (GQA), which allows for precise verification of the parameter density claims.

TII discloses the hardware used (1024 H100 GPU chips) for the training process. However, specific GPU-hours for the 3B variant's distillation and healing phase are not explicitly provided, nor is there a detailed carbon footprint calculation or energy consumption report specific to this model's development cycle.

TII provides benchmark results on standard sets like MMLU-Pro, MATH, and IFEval. They specify the use of the 'lm-evaluation-harness' framework. However, while they mention 'internal pipeline' settings, the exact prompts and few-shot configurations are not fully documented in a public technical report, leading to some community noted discrepancies when compared to standard leaderboard evaluations.

The model demonstrates high identity consistency, correctly identifying itself as part of the Falcon 3 family in its system prompts and documentation. There is no evidence of the model claiming to be a competitor's product (e.g., GPT-4), and it maintains a clear versioning identity across its base and instruct variants.

The model is released under the 'TII Falcon-LLM License 2.0'. This is a custom license based on Apache 2.0 but includes specific requirements, such as mandatory attribution for derivative works ('built using AI technology from TII'). While the terms are legally clear and allow for commercial use, it is not a standard OSI-approved license, which adds a layer of complexity for users.

Hardware requirements are well-documented, with specific VRAM estimates provided for various quantization levels (FP16, INT8, INT4). For example, FP16 is noted to require approximately 7-8GB of VRAM. The availability of official quantized versions (GGUF, AWQ, GPTQ) and documentation on their impact on memory makes the hardware footprint highly transparent.

The model follows a clear release versioning (Falcon 3 series), but a detailed, granular changelog for weight updates or specific 'drift' documentation is lacking. While the release date and initial version are clear, there is no established public system for tracking silent updates or performance changes over time beyond the initial Hugging Face commit history.