Qwen2-1.5B
Parameters
1.5B
Context Length
32.768K
Modality
Text
Architecture
Dense
License
Apache 2.0
Release Date
7 Jun 2024
Knowledge Cutoff
Sep 2024
Technical Specifications
Attention Structure
Grouped-Query Attention
Hidden Dimension Size
1536
Number of Layers
24
Attention Heads
32
Key-Value Heads
8
Activation Function
SwigLU
Normalization
RMS Normalization
Position Embedding
ROPE
System Requirements
VRAM requirements for different quantization methods and context sizes
Qwen2-1.5B
Qwen2-1.5B is a compact, decoder-only language model developed by the Qwen team at Alibaba Group. It is designed for efficient natural language processing tasks, striking a balance between performance and resource requirements. This model is a component of the broader Qwen2 series, which includes various model sizes and encompasses both base and instruction-tuned variants. Its purpose is to facilitate a wide array of applications that involve text generation, question answering, and comprehensive language understanding.
The architectural foundation of Qwen2-1.5B is the Transformer, incorporating several technical enhancements to optimize its operational characteristics. Key innovations include the integration of the SwiGLU activation function, the application of attention QKV bias, and the use of Group Query Attention (GQA). GQA contributes to more efficient inference processes and a reduced memory footprint during operation. The model also employs Rotary Positional Embeddings (RoPE) for handling positional information and utilizes RMSNorm for normalization. Furthermore, its tokenizer has undergone refinement, enabling adaptive processing of multiple natural languages and programming codes, which significantly expands its multilingual capabilities. Tied embeddings are used to enhance parameter efficiency within the model.
Regarding performance characteristics, Qwen2-1.5B exhibits robust capabilities across diverse language-centric tasks. It supports a context length of up to 32,768 tokens, allowing for the effective processing of extensive textual inputs. The model's functionalities span language understanding, text generation, code interpretation, mathematical problem-solving, and reasoning. Its design emphasizes efficiency and responsiveness, positioning it as a suitable selection for applications that necessitate rapid and reliable language processing across a multitude of languages.
About Qwen2
The Alibaba Qwen2 model family comprises large language models built upon the Transformer architecture. It includes both dense and Mixture-of-Experts (MoE) variants, designed for diverse language tasks. Technical features include Grouped Query Attention and support for extended context lengths up to 131,072 tokens, optimizing memory footprint for inference.
Other Qwen2 Models
Evaluation Benchmarks
Ranking is for Local LLMs.
No evaluation benchmarks for Qwen2-1.5B available.
Rankings
Overall Rank
-
Coding Rank
-
GPU Requirements
Full Calculator
Choose the quantization method for model weights
Context Size: 1,024 tokens