Mixtral-8x7B-v0.1
Active Parameters
46.7B
Context Length
33K
Modality
Text
Architecture
Mixture of Experts
KV Cache Rate
128 KB/tok
License
Apache 2.0
Release Date
9 Dec 2023
Knowledge Cutoff
Nov 2022
API Pricing (per 1M)
Input: $0.63 · Output: $0.63
Hardware & System Requirements
Precision
Context
Workload Memory: 28.51 GB
INT4 · 8K context
Base Weights: 25.01 GB
KV Cache: 1.00 GB
Runtime & Graphs: 2.50 GB
Minimum Required Hardware
1x NVIDIA A40 48GB
28.5 / 43.2 GB usable
2x RTX 4090 24GB
28.5 / 43.2 GB usable · TP 2
Apple M4 Max 128GB
28.5 / 96.0 GB usable
Determine requirements for your specific workload:
Open in VRAM Calculator
Runtime requirements
Checkpoint format
FP16
CUDA
>= 11.8
PyTorch
>= 2.1
Transformers
>= 4.36.0.dev0
| Architecture | Precision | Attention kernel |
|---|---|---|
Blackwell, Hopper | Native FP16 | FlashAttention-3 |
Ada Lovelace, Ampere | Native FP16 | FlashAttention-2 |
Architecture Diagram
Evaluation Benchmarks
Rank
#227
| Benchmark | Score | Rank |
|---|---|---|
General Text | 1197 | 187 |
Intelligence Index | 0.05 | 323 |
Rankings
Overall Rank
#227
Coding Rank
-
About Mixtral-8x7B-v0.1
Mixtral-8x7B-v0.1 is an open Sparse Mixture-of-Experts model from Mistral AI activating 12.9B parameters per token for high-efficiency inference. It delivers strong multilingual and coding performance across a 32K token context window.
Specifications
Attention
Attention Structure
Grouped-Query Attention
Attention Heads
32
Key-Value Heads
8
Attention Head Dimension
128
Position Embedding
ROPE
RoPE Theta
1,000,000
Normalization
RMS Normalization
Activation Function
Swish
Dimensions
Base Precision
FP16
Hidden Dimension Size
4,096
Number of Layers
32
FFN Intermediate Size (Dense)
14,336
Tokenizer
Vocabulary Size
32,000
Mixture of Experts
Total Expert Parameters
7.0B
Number of Experts
8
Active Experts
2
FFN Intermediate Size (per Expert)
14,336
About Mixtral
The Mixtral model family, developed by Mistral AI, employs a sparse Mixture-of-Experts (SMoE) architecture. This design utilizes multiple expert networks per layer, where a router selects a subset to process each token. This enables large total parameter counts while maintaining computational efficiency by activating only a fraction of parameters per forward pass.