Falcon2-11B
Parameters
11B
Context Length
8K
Modality
Text
Architecture
Dense
KV Cache Rate
20 KB/tok
License
TII Falcon License 2.0
Release Date
20 Jul 2024
Knowledge Cutoff
-
API Pricing (per 1M)
Self-hosted only
Hardware Sizing
Precision
Context
Workload Memory: 7.78 GB
INT4 · 8K context
Base Weights: 5.12 GB
KV Cache: 0.16 GB
Runtime & Graphs: 2.50 GB
Minimum Required Hardware
1x NVIDIA L4 24GB
7.8 / 21.6 GB usable
1x RTX 3060 12GB
7.8 / 10.8 GB usable
Apple M4 24GB
7.8 / 18.0 GB usable
Determine requirements for your specific workload:
Open in VRAM Calculator
Architecture Diagram
Evaluation Benchmarks
No evaluation benchmarks for Falcon2-11B available.
Rankings
Overall Rank
-
Coding Rank
-
About Falcon2-11B
Falcon 2 11B is an 11B parameter causal transformer developed by TII for multilingual text generation, translation, and domain fine-tuning. Pretrained on over 5T tokens, it features Grouped-Query Attention and an 8K token context window.
Technical Specifications
Attention
Attention Structure
Multi-Query Attention
Attention Heads
44
Key-Value Heads
1
Attention Head Dimension
128
Position Embedding
ROPE
RoPE Theta
500,042
Normalization
Layer Normalization
Activation Function
GELU
Dimensions
Hidden Dimension Size
5,632
Number of Layers
40
FFN Intermediate Size (Dense)
16,384
Tokenizer
Vocabulary Size
65,024
About Falcon 2
The Falcon 2 model family by TII encompasses the 11B language model and its Vision Language Model (VLM) counterpart. These open-source models, with 11 billion parameters, are trained on over five trillion tokens, providing multilingual support. The VLM variant integrates vision-to-language capabilities, enabling the processing of visual inputs for textual outputs.
Other Falcon 2 Models
- No related models available