GLM-4-9B

The GLM-4-9B architecture is extensively documented in the 'ChatGLM: A Family of Large Language Models' technical report and official GitHub repository. It clearly defines the model as a dense transformer using Grouped Query Attention (GQA) with 32 attention heads and 2 KV heads. Specific structural choices, such as the removal of bias terms (except in QKV layers) for length extrapolation and the use of RMSNorm and SwiGLU, are explicitly stated. The pre-training objective—an autoregressive blank-infilling task—is a well-documented departure from standard causal decoders, providing high transparency into its design lineage.

While the total token count (10 trillion) and general language support (26 languages) are disclosed, the specific composition of the dataset lacks granular detail. The technical report mentions the data is 'mostly Chinese and English' but does not provide a percentage breakdown or name specific sources beyond general categories. There is no public documentation on the filtering, cleaning, or deduplication methodologies used for the 10T token corpus, which is a significant gap for a model of this scale.

The tokenizer is fully transparent and publicly accessible via the 'tokenization_chatglm.py' script on Hugging Face. It uses a Tiktoken-based implementation with a clearly stated vocabulary size of 151,552 tokens. The code provides the exact regex patterns for tokenization and handles special tokens explicitly. This level of detail allows for full verification of how the model processes its claimed 26 supported languages.

The model is explicitly identified as a dense architecture with 9 billion total parameters. The technical report and configuration files confirm the layer count (40) and hidden dimension size (4096). While it is not a Mixture-of-Experts (MoE) model, the transparency regarding its parameter distribution across attention and feed-forward networks is high due to the availability of the configuration files and source code.

Transparency regarding training compute is very low. The technical report mentions that the model was trained with 'less training compute' than some competitors but fails to disclose specific GPU/TPU hours, hardware types used for the 10T token pre-training, or the duration of the training run. No information regarding the carbon footprint or estimated cost of training is provided.

The technical report provides scores for standard benchmarks (MMLU, GSM8K, HumanEval), but reproduction is hindered by limited disclosure of exact prompts and few-shot settings. While some evaluation code is available in the 'composite_demo' and 'LongAlign' repositories, third-party users have reported difficulties in matching official scores (e.g., on LongBench-Chat), suggesting gaps in the documented evaluation environment or sampling parameters.

The GLM-4-9B model demonstrates high identity consistency, correctly identifying itself as a product of Zhipu AI and the THUDM lab. It maintains clear versioning (e.g., distinguishing between the base 8K version and the 128K/1M context variants) and does not exhibit the identity confusion common in models that claim to be GPT-4 or other competitors.

The model weights are released under the MIT License, which is a highly permissive and clear open-source license. This is confirmed by official communications from the Z.ai organization on Hugging Face. However, there has been some historical ambiguity in repository documentation where Apache 2.0 and custom terms were mentioned, though the current official stance for the 9B weights is clearly MIT.

Hardware requirements are well-documented by both the developers and the community. Official documentation specifies VRAM requirements for inference (approx. 18-21GB for FP16) and provides guidance for quantization (INT4/INT8). The impact of context length on memory is also addressed, with specific mentions of YaRN scaling for extending the context window, though detailed memory-scaling curves are not provided.

The model uses a form of semantic versioning (e.g., GLM-4-9B-0414), and a changelog is maintained in the GitHub repository. However, updates can be irregular, and the transition between versions (such as the update requiring transformers >= 4.44.0) is documented primarily through README updates rather than a formal, centralized versioning system. There is limited public data on performance drift over time following safety or alignment updates.