Qwen2.5-1.5B

System Requirements

VRAM requirements for different quantization methods and context sizes

No evaluation benchmarks for Qwen2.5-1.5B available.

About Qwen2.5-1.5B

Qwen2.5-1.5B is a foundational large language model developed by Alibaba Cloud, forming part of the Qwen2.5 series. This model, with 1.54 billion parameters, is engineered for efficient processing and generation of human-like text across a diverse range of applications. It has undergone extensive pre-training on a large-scale dataset, encompassing up to 18 trillion tokens, and has been fine-tuned for specialized tasks such as instruction following, coding, and mathematical problem-solving. Its design emphasizes the ability to handle long contexts and generate coherent, accurate responses, making it suitable for various textual processing needs.

The architectural foundation of Qwen2.5-1.5B is a dense, decoder-only Transformer. Key components of its architecture include Rotary Position Embeddings (RoPE) for encoding positional information, SwiGLU as the activation function, and RMSNorm for effective normalization, which contribute to stable training and improved performance. The model incorporates Grouped Query Attention (GQA) with a specific configuration of 12 query heads and 2 key-value heads, facilitating efficient attention mechanisms. The model comprises 28 layers, with a hidden dimension size of 1536.

Qwen2.5-1.5B is designed to support a maximum context length of 128,000 tokens, with common configurations supporting 32,768 tokens for full context and enabling generation of up to 8,192 tokens. Its capabilities extend to multilingual understanding and generation across more than 29 languages. The model demonstrates proficiency in processing structured data formats such as tables and JSON. Practical use cases for Qwen2.5-1.5B include the development of conversational agents, virtual assistants, automated code generation tools, mathematical problem-solving platforms, and applications requiring robust content creation and summarization capabilities.

Technical Specifications

Resources

About Qwen2.5

Qwen2.5 by Alibaba is a family of dense, decoder-only language models available in various sizes, with some variants utilizing Mixture-of-Experts. These models are pretrained on large-scale datasets, supporting extended context lengths and multilingual communication. The family includes specialized models for coding, mathematics, and multimodal tasks, such as vision and audio processing.

Other Qwen2.5 Models

• Qwen2.5-0.5B
• Qwen2.5-3B
• Qwen2.5-7B
• Qwen2.5-14B
• Qwen2.5-32B
• Qwen2.5-72B