Technical Specifications

System Requirements

VRAM requirements for different quantization methods and context sizes

Qwen2.5-0.5B

Qwen2.5-0.5B is a foundational large language model developed by the Qwen team at Alibaba Cloud. It is part of the Qwen2.5 series, which represents an advancement in language model capabilities, featuring improvements in knowledge acquisition, coding proficiency, and mathematical reasoning. This variant, with approximately 0.49 billion parameters, serves as a robust base model, primarily designed for pretraining and subsequent fine-tuning for specialized applications. Its architecture is engineered to handle complex language tasks efficiently across multiple languages.

Architecturally, Qwen2.5-0.5B is a dense, decoder-only Transformer model. It incorporates Rotary Position Embedding (RoPE) for effective positional encoding, SwiGLU as its activation function, and RMSNorm for normalization. The attention mechanism utilizes Grouped Query Attention (GQA), specifically configured with 14 query heads and 2 key-value heads for this model size. The model is structured with 24 layers, contributing to its depth and capacity for learning intricate patterns in language data.

As a causal language model, Qwen2.5-0.5B is suitable for a range of downstream applications following post-training processes such as supervised fine-tuning or reinforcement learning from human feedback. Its capabilities include instruction following, generating extended text sequences, and processing structured data formats like JSON. The model supports a full context length of 32,768 tokens, with the broader Qwen2.5 series capable of handling contexts up to 128,000 tokens and generating outputs up to 8,000 tokens. It offers multilingual support, encompassing over 29 languages.

No evaluation benchmarks for Qwen2.5-0.5B available.