技术规格

系统要求

不同量化方法和上下文大小的显存要求

Falcon-40B

Falcon-40B is a 40-billion parameter causal decoder-only language model developed by the Technology Innovation Institute (TII). This foundational model was trained on one trillion tokens, primarily derived from the RefinedWeb dataset, which is a high-quality, filtered, and deduplicated web corpus, enhanced with additional curated data. The model's core objective is causal language modeling, which involves predicting the subsequent token in a given sequence. It is designed to serve as a robust base model for a variety of natural language processing applications.

The architectural design of Falcon-40B is an adaptation of the GPT-3 framework, incorporating specific modifications for enhanced efficiency and performance. Key architectural innovations include the implementation of rotary positional embeddings (RoPE) for improved handling of sequence positions, and an attention mechanism featuring both multiquery attention (MQA) and FlashAttention. MQA is a critical optimization, allowing for the sharing of a single key and value pair across all attention heads, thereby significantly improving inference scalability without impacting pretraining efficiency. The decoder block employs a parallel attention and Multi-Layer Perceptron (MLP) structure, augmented with two-layer normalization schemes to stabilize training and improve model performance.

Falcon-40B is optimized for efficient inference, which contributes to its higher processing speeds and scalability for deployment. As a raw, pretrained model, it is designed to be further fine-tuned for specific tasks. Its capabilities extend to various natural language generation and understanding applications, including content creation, machine translation, sentiment analysis, and language tutoring. The model supports several languages, exhibiting strong proficiency in English, German, Spanish, and French, alongside limited capabilities in Italian, Portuguese, Polish, Dutch, Romanian, Czech, and Swedish.

没有可用的 Falcon-40B 评估基准。