ApX logoApX logo

Phi-4

Parameters

14B

Context Length

16K

Modality

Text

Architecture

Dense

KV Cache Rate

160 KB/tok

License

MIT License

Release Date

13 Dec 2024

Knowledge Cutoff

Nov 2024

API Pricing (per 1M)

Input: $0.07 · Output: $0.14

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 10.27 GB

INT4 · 8K context

Base Weights: 6.52 GB

KV Cache: 1.25 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

10.3 / 21.6 GB usable

1x RTX 3060 12GB

10.3 / 10.8 GB usable

Apple M4 24GB

10.3 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.47.0

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 3.1k · Context: 16K · Vocab: 100.4kx 40 layersRMSNormPre-AttentionGrouped-Query Attention24Q / 8KV headsHead dim: 128+RMSNormPre-FFNFeed-Forward NetworkSwishIntermediate: 17.9k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#171

BenchmarkScoreRank

Professional Knowledge

MMLU Pro

0.704

42

Graduate-Level QA

GPQA

0.561

99

General Text

Text Arena

1256

179

Intelligence Index

Artificial Analysis

0.06

310

General Knowledge

Reference
MMLU

0.848

13

Rankings

Overall Rank

#171

Coding Rank

-

About Phi-4

Phi-4 is Microsoft's 14B parameter dense transformer model engineered with synthetic reasoning data for complex mathematics, logic, and scientific problem-solving. It delivers high-accuracy reasoning across a 16K token context window.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

24

Key-Value Heads

8

Position Embedding

ROPE

RoPE Theta

250,000

Normalization

RMS Normalization

Activation Function

Swish

Dimensions

Base Precision

FP16

Hidden Dimension Size

3,072

Number of Layers

40

FFN Intermediate Size (Dense)

17,920

Tokenizer

Vocabulary Size

100,352

About Phi-4

The Microsoft Phi-4 model family comprises small language models prioritizing efficient, high-capability reasoning. Its development emphasizes robust data quality and sophisticated synthetic data integration. This approach enables enhanced performance and on-device deployment capabilities.


Other Phi-4 Models