ApX logoApX logo

Phi-4 Reasoning Plus

Parameters

14B

Context Length

33K

Modality

Text

Architecture

Dense

KV Cache Rate

200 KB/tok

License

MIT

Release Date

30 Apr 2025

Knowledge Cutoff

Mar 2025

API Pricing (per 1M)

Self-hosted only

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 10.58 GB

INT4 · 8K context

Base Weights: 6.52 GB

KV Cache: 1.56 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

10.6 / 21.6 GB usable

1x RTX 3060 12GB

10.6 / 10.8 GB usable

Apple M4 24GB

10.6 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.51.1

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: AbsoluteHidden: 5.1k · Context: 33K · Vocab: 100.4kx 40 layersRMSNormPre-AttentionMulti-Head Attention40Q / 10KV headsHead dim: 128+RMSNormPre-FFNFeed-Forward NetworkSwiGLUIntermediate: 17.9k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#115

BenchmarkScoreRank

Professional Knowledge

MMLU Pro

0.76

35

Graduate-Level QA

GPQA

0.689

86

Rankings

Overall Rank

#115

Coding Rank

-

About Phi-4 Reasoning Plus

Phi-4 Reasoning Plus is an enhanced 14B parameter reasoning model by Microsoft trained with GRPO reinforcement learning for mathematical and scientific deduction. It generates explicit step-by-step reasoning traces across a 32K context window.

Specifications

Attention

Attention Structure

Multi-Head Attention

Attention Heads

40

Key-Value Heads

10

Position Embedding

Absolute Position Embedding

RoPE Theta

500,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

5,120

Number of Layers

40

FFN Intermediate Size (Dense)

17,920

Tokenizer

Vocabulary Size

100,352

About Phi-4

The Microsoft Phi-4 model family comprises small language models prioritizing efficient, high-capability reasoning. Its development emphasizes robust data quality and sophisticated synthetic data integration. This approach enables enhanced performance and on-device deployment capabilities.


Other Phi-4 Models