Falcon-7B

Falcon-7B is well-documented as a causal decoder-only transformer model. TII provides specific details on architectural modifications including Multi-Query Attention (MQA), FlashAttention, and Rotary Positional Embeddings (RoPE). The use of a parallel attention/MLP decoder block with a single layer normalization is explicitly detailed in the model card and the 'RefinedWeb' technical paper. While the training code 'Gigatron' is mentioned, it is not fully open-sourced, which prevents a perfect score.

The model's training data is extensively documented through the RefinedWeb paper and model card. TII discloses a breakdown of the 1,500B tokens: RefinedWeb-English (79%), Books (7%), Conversations (6%), Code (3%), RefinedWeb-French (3%), and Technical (2%). They also released a 600B token extract of RefinedWeb for public verification. However, the specific 'curated corpora' inspired by The Pile are described generally rather than with exhaustive file-level lists.

The tokenizer is publicly available on Hugging Face with a stated vocabulary size of 65,024 tokens. It uses a BPE-based approach and is shared across the Falcon-7B and 40B models. Documentation confirms it was trained on the RefinedWeb corpus, ensuring alignment with the pretraining data. Language support for English and French is explicitly stated and matches the tokenizer's design.

The model is clearly defined as a dense architecture with 7 billion total parameters. Detailed hyperparameters are provided: 32 layers, a hidden dimension of 4544, and 71 attention heads. Unlike MoE models, there is no ambiguity regarding active vs. total parameters, and the architectural breakdown is consistent across all official documentation.

TII discloses that the model was trained on 384 A100 40GB GPUs on AWS SageMaker over approximately two weeks. They provide the parallelism strategy (2D: PP=2, DP=192) and the custom 'Gigatron' codebase name. However, they do not provide a specific carbon footprint calculation or a detailed breakdown of total energy consumption in kWh, which are requirements for a high score in this category.

While Falcon-7B results are prominently featured on the Open LLM Leaderboard, the specific evaluation code and exact prompt templates used by TII for their internal reporting are not fully centralized in a single reproducible repository. The 'RefinedWeb' paper discusses evaluation philosophy but lacks a 'one-click' reproduction script for all claimed metrics. Discrepancies in scores across different leaderboard versions have been noted by the community.

Falcon-7B demonstrates high identity consistency. It correctly identifies itself as a model developed by TII and does not exhibit the 'identity crisis' seen in some models that claim to be GPT-4. The versioning is clear, and the model card explicitly distinguishes between the base and instruct variants.

The model is currently released under the Apache 2.0 license, which is a standard, well-understood open-source license allowing for commercial use without royalties. Although there was initial confusion regarding a custom TII license at launch, the transition to Apache 2.0 was swift and clearly documented across all official platforms (Hugging Face, TII website).

VRAM requirements are well-documented by both TII and the community, with approximately 15-16GB required for FP16 inference. Documentation for 4-bit and 8-bit quantization is available, noting that 4-bit can run on roughly 5.3GB of VRAM. While TII provides general guidance, detailed context-length scaling curves and specific accuracy-tradeoff benchmarks for various quantization levels are primarily provided by third parties rather than official documentation.

The model uses basic versioning on Hugging Face (e.g., commit hashes and tags), but lacks a formal, detailed changelog or semantic versioning for the weights themselves. While the release of 'Falcon-180B' and 'Falcon-2' followed, the original 7B model has not seen documented iterative updates that would allow users to track performance drift or safety tuning changes over time.