Falcon-3B

Falcon-3B is explicitly documented as a transformer-based causal decoder-only model. TII provides specific details on its derivation, noting it was pruned and 'healed' from the larger Falcon3-7B-Base model using knowledge distillation. Key architectural components are disclosed, including the use of Grouped Query Attention (GQA) with 12 query heads and 4 KV heads, SwiGLU activation, RMSNorm, and Rotary Positional Embeddings (RoPE) with a specific base value (1000042) to support its 32K context window. While a full peer-reviewed paper for the Falcon 3 series is less accessible than for Falcon 1, the technical specifications on Hugging Face and the official TII blog provide a clear lineage and structural breakdown.

The model's training involved a two-stage process: a large-scale pretraining of the 7B parent on 14 trillion tokens, followed by a 100-gigatoken 'healing' phase for the 3B variant. TII identifies the data categories as web, code, STEM, and multilingual content (English, French, Spanish, Portuguese). However, specific percentage breakdowns of these components are not provided, and the exact sources beyond the 'RefinedWeb' legacy are not publicly listed. The post-training dataset is described as 1.2 million samples covering STEM, conversations, and safety, but the specific datasets used for this alignment are not named or accessible for audit.

The tokenizer is publicly available via the Hugging Face repository and is well-documented. It features a vocabulary size of 131,072 tokens, which is a significant expansion over previous Falcon versions to improve compression and multilingual performance. The tokenizer approach is consistent with the claimed language support (EN, FR, ES, PT). Technical details such as the use of FlashAttention 2 for optimized computation are also verified in the model's configuration files.

The model is a dense architecture with 3 billion total parameters. TII provides a detailed architectural breakdown, including 22 decoder blocks, a hidden dimension of 1536, and a head dimension of 256. Unlike MoE models, there is no ambiguity regarding active vs. total parameters. The impact of quantization is also addressed through the official release of GGUF, AWQ, and 1.58-bit variants, providing transparency into how parameter density translates to different precision formats.

TII discloses the hardware used for the distillation and healing process (1024 H100 GPU chips). However, the total GPU hours for the 3B variant's specific training run are not explicitly stated, nor is there a calculated carbon footprint or detailed energy consumption report for this specific model. While the scale of the infrastructure is clear, the lack of duration and environmental metrics prevents a higher score.

TII provides scores for standard benchmarks like MMLU-PRO (29.7), MATH (19.9), and IFEval (54.4). They specify the use of the 'lm-evaluation-harness' and note that they report raw scores without 'fewshot_as_multiturn' to distinguish their results from competitors. However, the exact prompts, few-shot examples, and full evaluation code are not provided in a standalone reproducible repository, leading to reported discrepancies between internal TII scores and those on independent leaderboards like the Open LLM Leaderboard.

The model exhibits high identity consistency, correctly identifying itself as a member of the Falcon 3 family developed by TII. It does not suffer from the 'identity crisis' seen in some fine-tuned models that claim to be GPT-4 or Llama. Versioning is clear in the naming convention (Falcon3-3B-Instruct), and the model card explicitly outlines its intended use cases and limitations.

The model is released under the 'TII Falcon-LLM License 2.0'. This is a custom license based on Apache 2.0 but includes specific 'Acceptable Use' restrictions and requirements for attribution (e.g., 'built using Falcon LLM technology'). While the terms are publicly accessible and relatively clear, the use of a non-standard, custom license rather than a pure OSI-approved license like Apache 2.0 or MIT introduces some legal complexity for commercial users.

Hardware requirements are well-documented for various deployment scenarios. TII and community documentation provide VRAM estimates for FP16 (~7.3GB) and various quantized versions (INT8, INT4, 1.58-bit). The model is specifically marketed for consumer-grade hardware like laptops, and the documentation accurately reflects the memory scaling required for its 32K context window. The availability of multiple quantization formats (GGUF, AWQ) with associated performance notes aids transparency.

The model uses a clear naming convention for its initial release, but there is no public, centralized changelog or semantic versioning system to track updates to the weights or underlying datasets over time. While the release date (December 2024) is clear, the lack of a formal mechanism to notify users of silent updates or performance drift limits the score.