Falcon-40B
Parameters
40B
Context Length
2.048K
Modality
Text
Architecture
Dense
License
Apache 2.0
Release Date
5 Jun 2023
Knowledge Cutoff
Feb 2023
Technical Specifications
Attention Structure
Multi-Query Attention
Hidden Dimension Size
8192
Number of Layers
60
Attention Heads
64
Key-Value Heads
1
Activation Function
-
Normalization
Layer Normalization
Position Embedding
ROPE
System Requirements
VRAM requirements for different quantization methods and context sizes
Falcon-40B
Falcon-40B is a 40-billion parameter causal decoder-only language model developed by the Technology Innovation Institute (TII). This foundational model was trained on one trillion tokens, primarily derived from the RefinedWeb dataset, which is a high-quality, filtered, and deduplicated web corpus, enhanced with additional curated data. The model's core objective is causal language modeling, which involves predicting the subsequent token in a given sequence. It is designed to serve as a robust base model for a variety of natural language processing applications.
The architectural design of Falcon-40B is an adaptation of the GPT-3 framework, incorporating specific modifications for enhanced efficiency and performance. Key architectural innovations include the implementation of rotary positional embeddings (RoPE) for improved handling of sequence positions, and an attention mechanism featuring both multiquery attention (MQA) and FlashAttention. MQA is a critical optimization, allowing for the sharing of a single key and value pair across all attention heads, thereby significantly improving inference scalability without impacting pretraining efficiency. The decoder block employs a parallel attention and Multi-Layer Perceptron (MLP) structure, augmented with two-layer normalization schemes to stabilize training and improve model performance.
Falcon-40B is optimized for efficient inference, which contributes to its higher processing speeds and scalability for deployment. As a raw, pretrained model, it is designed to be further fine-tuned for specific tasks. Its capabilities extend to various natural language generation and understanding applications, including content creation, machine translation, sentiment analysis, and language tutoring. The model supports several languages, exhibiting strong proficiency in English, German, Spanish, and French, alongside limited capabilities in Italian, Portuguese, Polish, Dutch, Romanian, Czech, and Swedish.
About Falcon
The TII Falcon model family comprises causal decoder-only language models (7B, 40B). Their architecture, adapted from GPT-3, integrates rotary positional embeddings, Multi-Query Attention for inference efficiency, and FlashAttention for accelerated operations. Models are trained on the RefinedWeb dataset.
Other Falcon Models
Evaluation Benchmarks
Ranking is for Local LLMs.
No evaluation benchmarks for Falcon-40B available.
Rankings
Overall Rank
-
Coding Rank
-
GPU Requirements
Full Calculator
Choose the quantization method for model weights
Context Size: 1,024 tokens