Mellum 2.1 12B A2.5B Thinking

Active Parameters

12B

Context Length

131K

Modality

Text

Architecture

Mixture of Experts

KV Cache Rate

56 KB/tok

License

Apache 2.0

Release Date

20 Sept 2026

Knowledge Cutoff

-

API Pricing (per 1M)

Self-hosted only

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 9.69 GB

INT4 · 8K context

Base Weights: 6.75 GB

KV Cache: 0.44 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

9.7 / 21.6 GB usable

1x RTX 3060 12GB

9.7 / 10.8 GB usable

Apple M4 24GB

9.7 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 5.8.1

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 2.3k · Context: 131K · Vocab: 98.3kx 28 layersRMSNormPre-AttentionGrouped-Query Attention32Q / 4KV heads · SW: 1kHead dim: 128+RMSNormPre-FFNSparse MoE FFN (8/64 experts)SwiGLUIntermediate: 896+Final RMSNormOutput Logits

Evaluation Benchmarks

No evaluation benchmarks for Mellum 2.1 12B A2.5B Thinking available.

Rankings

Overall Rank

-

Coding Rank

-

About Mellum 2.1 12B A2.5B Thinking

Mellum 2.1 12B A2.5B Thinking is an MoE reasoning and coding model developed by JetBrains with 2.5B active parameters per token. It is tailored for advanced reasoning traces and software engineering workflows in IDE environments.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

32

Key-Value Heads

4

Attention Head Dimension

128

Position Embedding

ROPE

RoPE Theta

500,000

Sliding Window Attention

Yes

Sliding Window Size

1,024

Sliding Window Ratio

75.0%

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

2,304

Number of Layers

28

FFN Intermediate Size (Dense)

7,168

Tokenizer

Vocabulary Size

98,304

Mixture of Experts

Total Expert Parameters

2.5B

Number of Experts

64

Active Experts

8

Shared Experts

0

FFN Intermediate Size (per Expert)

896

Dense Layers Before MoE

0

About Mellum

The Mellum model family developed by JetBrains.


Other Mellum Models
  • No related models available