ApX logoApX logo

Falcon3-10B

Parameters

10B

Context Length

33K

Modality

Text

Architecture

Dense

KV Cache Rate

400 KB/tok

License

TII Falcon-LLM License 2.0

Release Date

17 Dec 2024

Knowledge Cutoff

Nov 2024

API Pricing (per 1M)

Self-hosted only

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 10.28 GB

INT4 · 8K context

Base Weights: 4.66 GB

KV Cache: 3.13 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

10.3 / 21.6 GB usable

1x RTX 3060 12GB

10.3 / 10.8 GB usable

Apple M4 24GB

10.3 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.46.1

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Evaluation Benchmarks

No evaluation benchmarks for Falcon3-10B available.

Rankings

Overall Rank

-

Coding Rank

-

About Falcon3-10B

Falcon3-10B is a 10B parameter causal language model from TII engineered for advanced scientific reasoning and conversational applications. It features Grouped-Query Attention and a 32K context window for detailed long-form text analysis.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

40

Key-Value Heads

10

Attention Head Dimension

256

Position Embedding

ROPE

RoPE Theta

1,000,042

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

5,120

Number of Layers

40

FFN Intermediate Size (Dense)

23,040

Tokenizer

Vocabulary Size

131,072

About Falcon 3

The TII Falcon 3 model family comprises open-source, decoder-only language models (1B-10B parameters) designed for efficiency. Key innovations include an extended 32K token context window, Grouped-Query Attention (GQA), and specialized versions for scientific and code-oriented applications. Some variants integrate Mamba-based architectures.


Other Falcon 3 Models