Mistral-Small-2501

Mistral Small 3 (2501) is clearly identified as a dense transformer model with a 24B parameter count. Key architectural details are publicly documented, including the use of 40 layers, a hidden dimension of 5120, 32 attention heads, and 8 KV heads (Grouped-Query Attention). It utilizes SwiGLU activation and Rotary Position Embeddings (RoPE) with a theta of 1M. While the high-level architecture is well-described in official blog posts and model cards, a full technical paper detailing the specific pre-training methodology and architectural innovations beyond standard transformer blocks is not provided.

Information regarding the training data is extremely limited. Mistral AI states that the model was not trained with Reinforcement Learning (RL) or synthetic data, positioning it as a 'pure' base model. However, there is no public disclosure of the specific data sources, the proportions of different data types (e.g., web, code, books), or the filtering and cleaning methodologies used. The claim of being 'knowledge-dense' is a marketing assertion without verifiable data provenance documentation.

The model uses the 'Tekken' tokenizer, which is publicly available via the mistral-common library. It features a large vocabulary size of 131,072 tokens, a significant increase from previous versions, designed to improve efficiency across dozens of supported languages. The tokenizer's performance and vocabulary are verifiable through the official GitHub repositories and integration with standard libraries like Hugging Face Transformers and vLLM.

The model's parameter count is explicitly stated as 24B (with some technical sources specifying 23.6B). As a dense model, all parameters are active during inference, which is clearly communicated. Detailed layer-wise specifications (40 layers, 32k hidden dim) are available through partner documentation (e.g., NVIDIA NIM, AWS), providing a clear breakdown of the model's density and structure.

There is virtually no public information regarding the compute resources used to train Mistral Small 3. No details on GPU/TPU hours, hardware clusters, training duration, or carbon footprint have been disclosed. The only compute-related information provided is for inference and fine-tuning requirements, not the original training phase.

Mistral provides benchmark results for standard evaluations like MMLU (81%), HumanEval, and GSM8K. They disclose that internal evaluation pipelines were used and acknowledge that results may vary from other reports. While they mention using third-party vendors for human evaluations, the exact prompts, few-shot examples, and evaluation code are not fully public, limiting the ability of independent researchers to exactly replicate the stated scores.

The model consistently identifies itself as Mistral Small 3 or the 2501 variant across official documentation, API endpoints, and model cards. It maintains a clear versioning identity (2501) to distinguish it from previous iterations (2409, 2402). There are no documented cases of the model misidentifying itself as a competitor's product or making misleading claims about its origin.

The model is released under the Apache 2.0 license, which is a highly permissive, standard open-source license. This applies to both the pre-trained base model and the instruction-tuned checkpoints. The terms are clear, allowing for both commercial and non-commercial use, modification, and distribution without conflicting proprietary restrictions.

Hardware requirements are well-documented for various deployment scenarios. Official sources specify that the model requires ~55GB of VRAM in BF16/FP16 but can be quantized to fit on a single 24GB GPU (like an RTX 4090) or a 32GB RAM MacBook. Detailed VRAM requirements for 4-bit quantization and its impact on accuracy (approx. 1.5 point drop) are available through community and partner documentation.

Mistral uses a date-based semantic versioning system (2501) and maintains a public changelog for its API and model releases. However, while new versions are clearly labeled, detailed documentation of behavioral drift or specific changes in weights between minor updates is less transparent. The transition from previous 'Small' versions to 2501 is noted, but a granular history of performance changes over time is not provided.