Mistral-7B-Instruct-v0.2

Parameters

7.3B

Context Length

33K

Modality

Text

Architecture

Dense

KV Cache Rate

128 KB/tok

License

Apache 2.0

Release Date

15 Jan 2024

Knowledge Cutoff

Dec 2023

API Pricing (per 1M)

Input: $0.20 · Output: $0.20

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 6.90 GB

INT4 · 8K context

Base Weights: 3.40 GB

KV Cache: 1.00 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

6.9 / 21.6 GB usable

1x RTX 3060 12GB

6.9 / 10.8 GB usable

Apple M4 24GB

6.9 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.36.0

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 4.1k · Context: 33K · Vocab: 32kx 32 layersRMSNormPre-AttentionGrouped-Query Attention32Q / 8KV headsHead dim: 128+RMSNormPre-FFNFeed-Forward NetworkSwishIntermediate: 14.3k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#244

BenchmarkScoreRank

General Text

Text Arena

1149

191

Rankings

Overall Rank

#244

Coding Rank

-

About Mistral-7B-Instruct-v0.2

Mistral-7B-Instruct-v0.2 is an enhanced open-weights instruction model from Mistral AI with full attention across an expanded 32K context window. It delivers reliable prompt adherence and low-latency text generation for general NLP workloads.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

32

Key-Value Heads

8

Position Embedding

ROPE

RoPE Theta

1,000,000

Normalization

RMS Normalization

Activation Function

Swish

Dimensions

Base Precision

FP16

Hidden Dimension Size

4,096

Number of Layers

32

FFN Intermediate Size (Dense)

14,336

Tokenizer

Vocabulary Size

32,000

About Mistral 7B

Mistral 7B, a 7.3 billion parameter model, utilizes a decoder-only transformer architecture. It features Sliding Window Attention and Grouped Query Attention for efficient long sequence processing. A Rolling Buffer Cache optimizes memory use, contributing to its design for efficient language processing.


Other Mistral 7B Models