ApX 标志

趋近智

GLM-4-9B

参数

9B

上下文长度

128K

模态

Text

架构

Dense

许可证

MIT License

发布日期

30 Jun 2024

知识截止

-

技术规格

注意力结构

Multi-Head Attention

隐藏维度大小

-

层数

40

注意力头

-

键值头

-

激活函数

-

归一化

RMS Normalization

位置嵌入

Absolute Position Embedding

系统要求

不同量化方法和上下文大小的显存要求

GLM-4-9B

The GLM-4-9B model, developed by THUDM (Tsinghua University Department of Computer Science and Technology) and Z.ai, represents an open-source iteration within the GLM-4 series of pre-trained language models. This model is engineered for general language tasks, exhibiting capabilities in semantic understanding, mathematical reasoning, code execution, and knowledge retrieval. It is designed to handle multilingual inputs and outputs, supporting 26 languages including Chinese, English, Japanese, Korean, and German. The GLM-4-9B also supports advanced functionalities such as web browsing, code execution, and custom tool calling through a Function Call mechanism.

Architecturally, GLM-4-9B employs a transformer architecture, which is a common deep learning structure for natural language processing. The model incorporates an autoregressive blank infilling approach for its pre-training phase. The architecture includes specific design choices such as the removal of bias terms except for those in the Query, Key, and Value (QKV) components of attention layers, which contributes to improved length extrapolation. It also utilizes RMSNorm for normalization. The base version of GLM-4-9B supports a context length of up to 128,000 tokens, with specialized variants offering an extended context length of up to 1 million tokens.

GLM-4-9B is engineered for a range of applications, including conversational AI assistants, content generation, and question answering systems. Its design facilitates integration with the Hugging Face Transformers library, simplifying deployment and adoption for developers. The model aims to provide a balance between efficiency and effectiveness, making it suitable for scenarios with resource constraints while maintaining performance across diverse tasks.

关于 GLM Family

General Language Models from Z.ai


其他 GLM Family 模型

评估基准

排名适用于本地LLM。

没有可用的 GLM-4-9B 评估基准。

排名

排名

-

编程排名

-

GPU 要求

完整计算器

选择模型权重的量化方法

上下文大小:1024 个令牌

1k
63k
125k

所需显存:

推荐 GPU