Falcon2-11B

The model's architecture is extensively documented in the official technical report and Hugging Face model card. It is a causal decoder-only transformer with specific modifications from GPT-3, including Rotary Positional Embeddings (RoPE), FlashAttention-2, and Grouped Query Attention (GQA) with 8 KV heads. The training methodology is detailed across four distinct stages, specifying context length increases (2048 to 8192) and the transition to high-quality curated data in the final stage.

TII provides a high-level breakdown of the 5.5 trillion token dataset, primarily citing RefinedWeb (English and European variants) along with code from 'The Stack' and curated conversational data. While the multi-stage data mixture is summarized in tables within the technical report, the exact proportions of the final 'high-quality' stage are less transparent, and the full dataset is not public, though the RefinedWeb component has separate public documentation.

The tokenizer is publicly accessible via the Hugging Face repository and is consistent with previous Falcon models. It has a stated vocabulary size of 65,024 tokens. Technical documentation confirms the use of a BPE-based approach, and the tokenizer's performance across the 11 supported languages is verifiable through the provided model files and evaluation results.

The model clearly states its 11 billion parameter count. As a dense model, all parameters are active during inference. Detailed architectural specifications are provided, including 60 transformer blocks, a hidden dimension of 4096, and 32 query heads, allowing for precise verification of the parameter density claims.

TII discloses that the model was trained on 1,024 NVIDIA A100 40GB GPUs using the Gigatron custom training codebase. The technical report mentions the use of 3D parallelism (TP=8, PP=1, DP=128) and ZeRO. While total GPU hours are not explicitly summed in a single figure, the hardware and parallelization strategy are detailed enough for independent estimation.

Evaluation results are provided for standard benchmarks like HellaSwag, MMLU, and ARC, with third-party verification from the Hugging Face Open LLM Leaderboard. However, the specific evaluation code and exact prompts used for internal testing are not fully public, and the technical report lacks a comprehensive reproduction guide for all claimed scores.

The model consistently identifies as a TII-developed foundation model. It does not exhibit significant identity confusion or claim to be a competitor's model in official documentation. It is transparent about being a raw pretrained model requiring further fine-tuning for specific tasks.

The model is released under the TII Falcon License 2.0. While based on Apache 2.0, it includes an 'Acceptable Use Policy' and specific terms regarding the 'Object' form of the model. The license is publicly available and clearly defines commercial use permissions, though the custom modifications from standard open-source licenses add a layer of legal complexity.

VRAM requirements are well-documented by both the provider and third-party sources (e.g., AWS documentation). It is noted that ~24GB is required for FP16 inference, and quantization impact (4-bit, 8-bit) is discussed in deployment guides. Memory scaling for the 8k context window is also addressed in technical specifications.

While the model is clearly versioned as 'Falcon 2', there is limited evidence of a formal public changelog or a structured system for tracking weight updates or performance drift over time. The transition from Falcon 1 to Falcon 2 is well-documented, but granular versioning for the 11B variant itself is minimal.