Llama 3.1 70B

Parameters

70B

Context Length

128K

Modality

Text

Architecture

Dense

KV Cache Rate

320 KB/tok

License

Llama 3.1 Community License Agreement

Release Date

23 Jul 2024

Knowledge Cutoff

Dec 2023

API Pricing (per 1M)

Input: $0.72 · Output: $0.72

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 37.60 GB

INT4 · 8K context

Base Weights: 32.60 GB

KV Cache: 2.50 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA A40 48GB

37.6 / 43.2 GB usable

2x RTX 4090 24GB

37.6 / 43.2 GB usable · TP 2

Apple M4 Max 128GB

37.6 / 96.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.40.1

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 8.2k · Context: 128Kx 80 layersNormPre-AttentionGrouped-Query Attention64Q / 8KV headsHead dim: 128+NormPre-FFNFeed-Forward NetworkActivation+Final NormOutput Logits

Evaluation Benchmarks

Rank

#186

BenchmarkScoreRank

Professional Knowledge

MMLU Pro

0.664

50

Graduate-Level QA

GPQA

0.417

115

General Text

Text Arena

1293

171

Intelligence Index

Artificial Analysis

0.07

301

General Knowledge

Reference
MMLU

0.836

15

Summarization

Archived
ProLLM Summarization

0.598

23

Rankings

Overall Rank

#186

Coding Rank

-

About Llama 3.1 70B

Llama 3.1 70B is a high-performance open language model from Meta built for advanced coding assistance, multilingual conversation, and enterprise RAG. It supports long-sequence comprehension and tool integration across a 128K context window.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

64

Key-Value Heads

8

Position Embedding

ROPE

Dimensions

Base Precision

FP16

Hidden Dimension Size

8,192

Number of Layers

80

About Llama 3.1

Llama 3.1 is Meta's advanced large language model family, building upon Llama 3. It features an optimized decoder-only transformer architecture, available in 8B, 70B, and 405B parameter versions. Significant enhancements include an expanded 128K token context window and improved multilingual capabilities across eight languages, refined through data and post-training procedures.


Other Llama 3.1 Models