Phi-1.5

Parameters

1.3B

Context Length

2K

Modality

Text

Architecture

Dense

KV Cache Rate

192 KB/tok

License

MIT

Release Date

10 Sept 2023

Knowledge Cutoff

-

API Pricing (per 1M)

Self-hosted only

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 3.48 GB

INT4 · 2048 context

Base Weights: 0.61 GB

KV Cache: 0.38 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

3.5 / 21.6 GB usable

1x RTX 3060 12GB

3.5 / 10.8 GB usable

Apple M4 24GB

3.5 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.37.0

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 2k · Context: 2K · Vocab: 51.2kx 24 layersRMSNormPre-AttentionMulti-Head Attention32Q / 32KV headsHead dim: 64+RMSNormPre-FFNFeed-Forward NetworkGELUIntermediate: 8.2k+Final RMSNormOutput Logits

Evaluation Benchmarks

No evaluation benchmarks for Phi-1.5 available.

Rankings

Overall Rank

-

Coding Rank

-

About Phi-1.5

Phi-1.5 is a 1.3B parameter open language model from Microsoft Research engineered for common sense reasoning and general natural language understanding. It was trained on 30B high-quality synthetic tokens for efficient NLP research.

Specifications

Attention

Attention Structure

Multi-Head Attention

Attention Heads

32

Key-Value Heads

32

Position Embedding

ROPE

RoPE Theta

10,000

Normalization

RMS Normalization

Activation Function

GELU

Dimensions

Base Precision

FP16

Hidden Dimension Size

2,048

Number of Layers

24

FFN Intermediate Size (Dense)

8,192

Tokenizer

Vocabulary Size

51,200

About Phi-1.5

Microsoft's Phi-1.5 is a 1.3 billion parameter Transformer model, a successor to Phi-1. It was trained on a curated synthetic dataset of "textbook-quality" for common sense reasoning. The architecture comprises 24 layers, 32 attention heads, and incorporates rotary embeddings.


Other Phi-1.5 Models
  • No related models available