Falcon-1B
Parameters
1B
Context Length
8K
Modality
Text
Architecture
Dense
KV Cache Rate
2 KB/tok
License
TII Falcon-LLM License 2.0
Release Date
17 Dec 2024
Knowledge Cutoff
-
API Pricing (per 1M)
Self-hosted only
Hardware Sizing
Precision
Context
Workload Memory: 2.98 GB
INT4 · 8K context
Base Weights: 0.47 GB
KV Cache: 0.02 GB
Runtime & Graphs: 2.50 GB
Minimum Required Hardware
1x NVIDIA L4 24GB
3.0 / 21.6 GB usable
1x RTX 3060 12GB
3.0 / 10.8 GB usable
Apple M4 24GB
3.0 / 18.0 GB usable
Determine requirements for your specific workload:
Open in VRAM Calculator
Architecture Diagram
Evaluation Benchmarks
No evaluation benchmarks for Falcon-1B available.
Rankings
Overall Rank
-
Coding Rank
-
About Falcon-1B
Falcon3-1B is an open 1B parameter causal transformer developed by TII for scientific reasoning, mathematics, and edge AI applications. It incorporates Grouped-Query Attention and SwiGLU activations for efficient multilingual inference.
Technical Specifications
Attention
Attention Structure
Multi-Query Attention
Attention Heads
32
Key-Value Heads
1
Position Embedding
ROPE
Normalization
RMS Normalization
Activation Function
SwigLU
Dimensions
Hidden Dimension Size
768
Number of Layers
24
About Falcon
The TII Falcon model family comprises causal decoder-only language models (7B, 40B). Their architecture, adapted from GPT-3, integrates rotary positional embeddings, Multi-Query Attention for inference efficiency, and FlashAttention for accelerated operations. Models are trained on the RefinedWeb dataset.