Falcon3-7B

Falcon 3-7B is explicitly documented as a transformer-based causal decoder-only model with 28 decoder blocks. TII provides specific architectural details including the use of Grouped Query Attention (GQA) with 12 query heads and 4 key-value heads, a head dimension of 256, and SwiGLU activation with RMSNorm. The use of Rotary Positional Embedding (RoPE) with a specific base (1000042) to support its 32K context window is well-documented in official model cards and technical blogs. While a full peer-reviewed paper for the Falcon 3 iteration is pending (announced for early 2025), the technical specifications are highly detailed and publicly accessible via Hugging Face and TII's official portal.

TII discloses that the model was pretrained on 14 trillion tokens and post-trained on 1.2 million samples. General categories are provided: web data, code, STEM, and multilingual content (English, French, Spanish, Portuguese). However, specific percentage breakdowns of these sources are not publicly available. While TII has a history of releasing data extracts (e.g., RefinedWeb for Falcon 1), the specific composition and filtering methodology for the 14T token Falcon 3 dataset remain described in general terms rather than granular detail.

The tokenizer is fully accessible via the Hugging Face repository and integrated into the `transformers` library. It features a clearly stated vocabulary size of 131,072 tokens, which is a significant increase from previous versions to improve compression across its supported languages. The tokenization approach is verifiable through the public `tokenizer_config.json` and `tokenizer.json` files, which confirm the claimed multilingual support and special token handling (e.g., chat templates).

The model is a dense architecture with approximately 7.46 billion total parameters. Unlike MoE models, all parameters are active during inference, which is clearly reflected in the documentation and model files. The architectural breakdown (layers, heads, dimensions) is precisely defined, allowing for independent verification of the parameter count based on the provided configuration files.

TII has disclosed that the 7B model was trained using 1024 H100 GPU chips (some sources mention 2048). While the hardware type and scale are provided, the exact training duration in hours and the total carbon footprint or energy consumption metrics are not explicitly detailed in the current release documentation. This provides a moderate level of transparency compared to models that withhold all compute data, but falls short of exemplary reporting.

TII provides benchmark results on standard sets like MMLU, MMLU-Pro, ARC, and GSM8K. They specify the use of the `lm-evaluation-harness` and provide raw scores. However, while they mention the evaluation methodology (e.g., 5-shot), the exact evaluation code and specific prompt templates used to achieve the reported scores are not bundled in a single reproducible repository, requiring users to rely on the general harness settings.

The model consistently identifies itself as Falcon 3 in its system prompts and documentation. It does not exhibit the identity confusion seen in some fine-tuned models that claim to be GPT-4 or Llama. Versioning is clear within the 'Falcon 3' family, distinguishing between Base, Instruct, and Mamba variants.

The model is released under the 'TII Falcon-LLM License 2.0'. This is a custom license based on Apache 2.0 but includes specific 'Acceptable Use' restrictions and requirements for derivative works. While the terms are publicly available and clearly written, it is not a standard OSI-approved open-source license, which introduces some complexity for commercial users compared to a pure Apache 2.0 or MIT license.

VRAM requirements are well-documented for various precisions. Official and third-party sources (like Ollama and Hugging Face) provide clear guidance: ~15GB for FP16 and ~4.6GB for 4-bit quantization. TII also explicitly supports and documents various quantization formats (GGUF, AWQ, GPTQ) and their impact on deployment, making the hardware footprint highly predictable for end-users.

Falcon 3 uses a family-based naming convention (1B, 3B, 7B, 10B) and distinguishes between Base and Instruct versions. However, there is no formal semantic versioning (e.g., v3.1.0) or a detailed public changelog for minor weight updates or 'silent' refreshes. Users must rely on Hugging Face commit histories to track changes, which lacks the transparency of a formal release management system.