GLM-130B
Parameters
130B
Context Length
2K
Modality
Text
Architecture
Dense
License
Apache 2.0
Release Date
4 Aug 2022
Knowledge Cutoff
Jul 2022
API Pricing (per 1M)
Self-hosted only
System Requirements
VRAM requirements for different quantization methods and context sizes
1,024 tokens
Consumer
14x RTX 4090
24GB VRAM
Datacenter
4x NVIDIA A100
80GB VRAM
Apple Silicon
3x Apple M3 Max
128GB VRAM
2,048 tokens
Consumer
14x RTX 4090
24GB VRAM
Datacenter
4x NVIDIA A100
80GB VRAM
Apple Silicon
3x Apple M3 Max
128GB VRAM
Architecture Diagram
Evaluation Benchmarks
No evaluation benchmarks for GLM-130B available.
Rankings
Overall Rank
-
Coding Rank
-
About GLM-130B
GLM-130B is a bidirectional bilingual foundation model developed by Tsinghua KEG and Zhipu AI for English and Chinese language comprehension. Built with DeepNorm and RoPE, it enables efficient large-scale inference and blank-infilling generation.
Technical Specifications
Attention
Attention Structure
Multi-Head Attention
Attention Heads
-
Key-Value Heads
-
Attention Head Dimension
-
Position Embedding
Absolute Position Embedding
RoPE Theta
-
Sliding Window Attention
-
Sliding Window Size
-
Sliding Window Ratio
-
Linear Attention
-
Linear Attention Ratio
-
Normalization
Deep Normalization
Activation Function
GELU
Dimensions
Hidden Dimension Size
12,288
Number of Layers
70
FFN Intermediate Size (Dense)
-
Multi-Token Prediction Heads
-
Tokenizer
Vocabulary Size
-
About GLM Family
General Language Models from Z.ai