Falcon2-11B
Parameters
11B
Context Length
8K
Modality
Text
Architecture
Dense
License
TII Falcon License 2.0
Release Date
20 Jul 2024
Knowledge Cutoff
-
API Pricing (per 1M)
Self-hosted only
System Requirements
VRAM requirements for different quantization methods and context sizes
1,024 tokens
Consumer
2x RTX 4090
24GB VRAM
Datacenter
1x NVIDIA A100
80GB VRAM
Apple Silicon
1x Apple M3 Max
128GB VRAM
8,192 tokens
Consumer
2x RTX 4090
24GB VRAM
Datacenter
1x NVIDIA A100
80GB VRAM
Apple Silicon
1x Apple M3 Max
128GB VRAM
Architecture Diagram
Evaluation Benchmarks
No evaluation benchmarks for Falcon2-11B available.
Rankings
Overall Rank
-
Coding Rank
-
About Falcon2-11B
Falcon 2 11B is an 11B parameter causal transformer developed by TII for multilingual text generation, translation, and domain fine-tuning. Pretrained on over 5T tokens, it features Grouped-Query Attention and an 8K token context window.
Technical Specifications
Attention
Attention Structure
Multi-Query Attention
Attention Heads
44
Key-Value Heads
1
Attention Head Dimension
128
Position Embedding
ROPE
RoPE Theta
500,042
Sliding Window Attention
No
Sliding Window Size
-
Sliding Window Ratio
-
Linear Attention
-
Linear Attention Ratio
-
Normalization
Layer Normalization
Activation Function
GELU
Dimensions
Hidden Dimension Size
5,632
Number of Layers
40
FFN Intermediate Size (Dense)
16,384
Multi-Token Prediction Heads
-
Tokenizer
Vocabulary Size
65,024
About Falcon 2
The Falcon 2 model family by TII encompasses the 11B language model and its Vision Language Model (VLM) counterpart. These open-source models, with 11 billion parameters, are trained on over five trillion tokens, providing multilingual support. The VLM variant integrates vision-to-language capabilities, enabling the processing of visual inputs for textual outputs.
Other Falcon 2 Models
- No related models available