Mixtral-8x22B-v0.1

Active Parameters

176B

Context Length

66K

Modality

Text

Architecture

Mixture of Experts

KV Cache Rate

37 KB/tok

License

Apache 2.0

Release Date

10 Apr 2024

Knowledge Cutoff

-

API Pricing (per 1M)

Input: $0.90 · Output: $0.90

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 371.59 GB

FP16 · 8K context

Base Weights: 368.80 GB

KV Cache: 0.29 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

8x NVIDIA H100 80GB

371.6 / 576.0 GB usable · TP 8

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.38.0

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 1k · Context: 66K · Vocab: 32kx 56 layersRMSNormPre-AttentionGrouped-Query Attention48Q / 8KV headsHead dim: 21+RMSNormPre-FFNSparse MoE FFN (2/8 experts)SwiGLUIntermediate: 16.4k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#220

BenchmarkScoreRank

General Text

Text Arena

1230

181

Intelligence Index

Artificial Analysis

0.06

310

Summarization

Archived
ProLLM Summarization

0.587

25

Rankings

Overall Rank

#220

Coding Rank

-

About Mixtral-8x22B-v0.1

Mixtral-8x22B-v0.1 is Mistral AI's 176B Sparse Mixture-of-Experts model activating 39B parameters for complex mathematical, coding, and multilingual tasks. It features native function calling and high computational throughput across a 64K context window.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

48

Key-Value Heads

8

Position Embedding

ROPE

RoPE Theta

1,000,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

1,024

Number of Layers

56

FFN Intermediate Size (Dense)

16,384

Tokenizer

Vocabulary Size

32,000

Mixture of Experts

Total Expert Parameters

22.0B

Number of Experts

8

Active Experts

2

FFN Intermediate Size (per Expert)

16,384

About Mixtral

The Mixtral model family, developed by Mistral AI, employs a sparse Mixture-of-Experts (SMoE) architecture. This design utilizes multiple expert networks per layer, where a router selects a subset to process each token. This enables large total parameter counts while maintaining computational efficiency by activating only a fraction of parameters per forward pass.


Other Mixtral Models