Phi-3-medium

Parameters

14B

Context Length

128K

Modality

Text

Architecture

Dense

KV Cache Rate

200 KB/tok

License

MIT

Release Date

22 Apr 2024

Knowledge Cutoff

Oct 2023

API Pricing (per 1M)

Input: $0.17 · Output: $0.68

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 10.58 GB

INT4 · 8K context

Base Weights: 6.52 GB

KV Cache: 1.56 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

10.6 / 21.6 GB usable

1x RTX 3060 12GB

10.6 / 10.8 GB usable

Apple M4 24GB

10.6 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.39.3

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Additional required packages

Install alongside standard runtime libraries:

pip install flash_attn

Architecture Diagram

Evaluation Benchmarks

Rank

#241

BenchmarkScoreRank

General Text

Text Arena

1198

187

Rankings

Overall Rank

#241

Coding Rank

-

About Phi-3-medium

Phi-3-medium is Microsoft's 14B parameter dense foundation model delivering advanced reasoning and code synthesis in compute-constrained environments. Optimized for ONNX Runtime, it supports extended context lengths up to 128K tokens.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

40

Key-Value Heads

10

Position Embedding

ROPE

RoPE Theta

10,000

Sliding Window Attention

Yes

Sliding Window Size

2,047

Normalization

RMS Normalization

Activation Function

Swish

Dimensions

Base Precision

FP16

Hidden Dimension Size

5,120

Number of Layers

40

FFN Intermediate Size (Dense)

17,920

Tokenizer

Vocabulary Size

32,064

About Phi-3

Microsoft's Phi-3 models are small language models designed for efficient operation on resource-constrained devices. They utilize a transformer decoder architecture and are trained on extensively filtered, high-quality data, including synthetic compositions. This approach enables a compact yet capable model family.


Other Phi-3 Models