ApX logoApX logo

GLM-4

Parameters

32B

Context Length

128K

Modality

Text

Architecture

Dense

KV Cache Rate

61 KB/tok

License

Custom Commercial License with Restrictions

Release Date

15 Jan 2024

Knowledge Cutoff

Dec 2023

API Pricing (per 1M)

Input: $0.44 · Output: $1.76

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 17.88 GB

INT4 · 8K context

Base Weights: 14.90 GB

KV Cache: 0.48 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

17.9 / 21.6 GB usable

2x RTX 4090 24GB

17.9 / 43.2 GB usable · TP 2

Apple M4 24GB

17.9 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

FP16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.44.0

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native FP16

FlashAttention-3

Ada Lovelace, Ampere

Native FP16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: AbsoluteHidden: 6.1k · Context: 128K · Vocab: 151.6kx 61 layersRMSNormPre-AttentionMulti-Head Attention48Q / 2KV headsHead dim: 128+RMSNormPre-FFNFeed-Forward NetworkSwiGLUIntermediate: 13.7k+Final RMSNormOutput Logits

Evaluation Benchmarks

Rank

#172

BenchmarkScoreRank

General Text

Text Arena

1319

161

Rankings

Overall Rank

#172

Coding Rank

-

About GLM-4

GLM-4 32B is a dense foundation model by Z.ai engineered for engineering-grade code generation, structured outputs, and bilingual reasoning. Trained on 15T tokens, it provides a stable engine for autonomous tool execution across a 128K context window.

Specifications

Attention

Attention Structure

Multi-Head Attention

Attention Heads

48

Key-Value Heads

2

Attention Head Dimension

128

Position Embedding

Absolute Position Embedding

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

FP16

Hidden Dimension Size

6,144

Number of Layers

61

FFN Intermediate Size (Dense)

13,696

Tokenizer

Vocabulary Size

151,552