Ministral 3 3B
Parameters
3B
Context Length
256K
Modality
Multimodal
Architecture
Dense
License
Apache 2.0
Release Date
2 Dec 2025
Knowledge Cutoff
-
System Requirements
VRAM requirements for different quantization methods and context sizes
1,024 tokens
Consumer
1x RTX 4090
24GB VRAM
Datacenter
1x NVIDIA A100
80GB VRAM
Apple Silicon
1x Apple M3 Max
128GB VRAM
256,000 tokens
Consumer
2x RTX 4090
24GB VRAM
Datacenter
1x NVIDIA A100
80GB VRAM
Apple Silicon
1x Apple M3 Max
128GB VRAM
Architecture Diagram
Evaluation Benchmarks
Rank
#123
| Benchmark | Score | Rank |
|---|---|---|
General Knowledge | 0.707 | 29 |
Rankings
Overall Rank
#123
Coding Rank
-
About Ministral 3 3B
Ministral 3 3B is a compact, multimodal language model engineered by Mistral AI for efficient execution in edge computing environments and resource-constrained scenarios. The model architecture integrates a 3.4 billion parameter language decoder with a 410 million parameter Vision Transformer (ViT) encoder, yielding a combined capacity of approximately 3.8 billion parameters. This hybrid design enables the simultaneous processing of text and visual inputs, facilitating advanced tasks such as image captioning, visual question answering, and multimodal data extraction while maintaining a low computational overhead.
Technically, Ministral 3 3B follows a dense Transformer-based decoder-only architecture that leverages Grouped Query Attention (GQA) with 32 query heads and 8 key-value heads to optimize memory bandwidth and inference speed. It employs Rotary Positional Embeddings (RoPE) enhanced with YaRN (Yet another RoPE extensioN) and position-based softmax temperature scaling to support an extensive context window of up to 256,000 tokens. To further enhance efficiency at this scale, the 3B variant utilizes tied input-output embeddings, preventing vocabulary parameters from disproportionately increasing the total model size. The vision component utilizes a frozen ViT encoder derived from the Mistral Small 3.1 architecture, coupled with a newly trained multimodal projection layer.
The model is optimized for high-performance on-device applications, offering native support for function calling and structured JSON output to enable complex agentic workflows. It incorporates architectural refinements such as SwiGLU activation and RMSNorm to ensure stability and efficiency during local inference. By supporting dozens of languages and featuring a high-context capacity, Ministral 3 3B is positioned as a versatile solution for real-time translation, local content generation, and privacy-focused intelligent assistants operating directly on user hardware.
Technical Specifications
Attention
Attention Structure
Multi-Head Attention
Attention Heads
32
Key-Value Heads
8
Attention Head Dimension
128
Position Embedding
Absolute Position Embedding
RoPE Theta
1,000,000
Sliding Window Attention
No
Sliding Window Size
-
Sliding Window Ratio
-
Linear Attention
-
Linear Attention Ratio
-
Normalization
Layer Normalization
Activation Function
SwigLU
Dimensions
Hidden Dimension Size
3,072
Number of Layers
26
FFN Intermediate Size (Dense)
9,216
Multi-Token Prediction Heads
-
Tokenizer
Vocabulary Size
131,072
About Ministral 3
Ministral 3 is a family of efficient edge models with vision capabilities, available in 3B, 8B, and 14B parameter sizes. Designed for edge deployment with multimodal and multilingual support, offering best-in-class performance for resource-constrained environments.