Falcon-180B

The model's architecture is extensively documented in the technical paper 'The Falcon Series of Open Language Models' and official model cards. It is a causal decoder-only transformer that evolves from the Falcon-40B, incorporating specific modifications such as Multi-Query Attention (MQA) for inference efficiency, Rotary Position Embeddings (RoPE), and FlashAttention. The use of parallel attention/MLP blocks with two layer norms is explicitly detailed. While the pretraining procedure is described as using the custom 'Gigatron' codebase with 3D parallelism, the codebase itself is not fully open-sourced, preventing a perfect score.

TII provides a high-level breakdown of the 3.5 trillion token training set: 85% RefinedWeb, 3% code, and 12% curated data (conversations and technical papers). The RefinedWeb dataset methodology is publicly documented in a separate paper, and a 600B token extract has been released for public audit. However, the specific 'curated' portions (conversations and technical papers) lack granular source lists or public availability, and the exact filtering criteria for the 180B run specifically are less detailed than the general RefinedWeb documentation.

The Falcon tokenizer is publicly available via Hugging Face and is well-documented. It uses a vocabulary size of 65,024 tokens, optimized for multilingual support (English, German, Spanish, French, etc.). The technical report specifies the use of 16-bit unsigned integers for storage and notes the inclusion of extra values for downstream adaptation. The tokenizer is consistent across the Falcon family, facilitating cross-model verification.

The model is a dense architecture with 180 billion parameters. Technical specifications are highly detailed, including 80 layers, a hidden dimension of 14,848, and 64 attention heads. Because it is a dense model, 'active parameters' equals total parameters, and this is clearly communicated. The architectural breakdown (layers, heads, dimensions) is provided in official documentation and verified by the model weights on Hugging Face.

Training compute is disclosed with significant detail: approximately 7,000,000 GPU hours on up to 4,096 NVIDIA A100-40GB GPUs. The infrastructure (AWS SageMaker P4d instances) and the 3D parallelism strategy (TP=8, PP=8, DP=64) are explicitly stated. While the specific carbon footprint calculation is not provided in the primary model card, the hardware and duration data allow for independent estimation.

TII reports results on standard benchmarks (MMLU, HellaSwag, ARC, etc.) and provides some 1-shot/0-shot context in their technical paper. However, the exact evaluation code and full prompt templates used for the internal results are not as transparently shared as the model weights. While the model's performance is verifiable via the Open LLM Leaderboard, the lack of a dedicated, public reproduction repository for the paper's specific figures limits the score.

The model consistently identifies itself as Falcon-180B and is transparent about its nature as a foundation model. The distinction between the base model and the 'Chat' variant (fine-tuned on Ultrachat, Platypus, and Airoboros) is clearly maintained in all documentation. There are no known instances of the model claiming a competitor's identity or misrepresenting its origin.

The model uses a bespoke 'Falcon-180B TII License'. While it allows commercial use, it is not a standard OSI-approved license. It contains significant restrictions, particularly regarding 'Hosting Use' (requiring a separate agreement for API/managed services) and an Acceptable Use Policy that can be updated 'from time to time'. The complexity of these custom terms creates ambiguity for commercial users compared to standard licenses like Apache 2.0.

Hardware requirements are well-documented by both TII and the community. Official guidance states a requirement of ~400GB VRAM for FP16 inference, typically necessitating 8x A100 (80GB) GPUs. Quantization impacts are documented, with 4-bit (INT4) requirements noted at ~100GB. Third-party guides (e.g., RunPod, Vast.ai) provide extensive verification of these requirements, though TII's own documentation could be more granular regarding context-length memory scaling.

The model follows a basic versioning scheme (Falcon-180B and Falcon-180B-Chat), but there is no formal, public changelog or semantic versioning for weight updates or minor iterations. While the release date is clear, the long-term maintenance and tracking of potential silent updates or drift are not supported by a robust versioning infrastructure.