DeepSeek-V3 671B

DeepSeek-V3 provides exemplary architectural transparency through a detailed technical report and open-source implementation. The model explicitly documents its use of Multi-head Latent Attention (MLA) for inference efficiency and the DeepSeekMoE architecture. It provides specific details on its novel auxiliary-loss-free load balancing strategy and Multi-Token Prediction (MTP) objective. The transition from previous versions (V2) is clearly documented, and the model's 61-layer decoder-only transformer structure is fully specified in both the paper and the public GitHub repository.

While the total token count (14.8 trillion) and the general nature of the data (diverse, high-quality, multilingual) are disclosed, there is a lack of granular detail regarding the specific dataset proportions or sources. The documentation mentions 'web, code, and math' but does not provide a percentage breakdown or specific filtering/cleaning methodologies beyond general claims of curation. No sample data or specific source lists are publicly available, making the composition difficult to verify independently.

The tokenizer is publicly accessible via Hugging Face and GitHub, with a confirmed vocabulary size of 129,280 tokens. It uses a byte-level Byte-Pair Encoding (BPE) approach similar to the Llama tokenizer but with custom modifications for multilingual support (English and Chinese). Documentation includes specific special tokens for tool calling and reasoning blocks, and the vocabulary is consistent across the V3 family, ensuring predictable behavior for developers.

DeepSeek-V3 is transparent about its Mixture-of-Experts (MoE) structure, clearly stating a total of 671 billion parameters with 37 billion active parameters per token. The architectural breakdown (256 routed experts, 1 shared expert) is well-documented. However, it loses points because the 671B figure includes 14B parameters from the Multi-Token Prediction (MTP) module which are used during training but are optional/detachable during inference, leading to some minor ambiguity in 'total' vs 'inference' parameter counts in marketing materials.

The technical report provides unusually specific details on training compute, citing 2.788 million H800 GPU hours for the full training run. It discloses the hardware used (2,048 NVIDIA H800 GPUs), the training duration (approximately two months), and even provides a cost estimate (~$5.58 million). While it does not provide a formal carbon footprint calculation in the primary report, the level of compute transparency is significantly higher than most industry peers.

DeepSeek provides a comprehensive list of benchmark results (MMLU, GSM8K, HumanEval, etc.) in its technical report and GitHub. However, it lacks a unified, one-click reproduction script for all claimed figures. While evaluation settings (e.g., few-shot counts) are mentioned, the exact prompts and internal evaluation pipelines are not fully open-sourced, making exact bit-for-bit reproduction of scores challenging for independent auditors.

The model generally maintains a consistent identity as DeepSeek-V3 and correctly identifies its version and origin in most standard deployments. It is transparent about its nature as an AI and its MoE architecture. Some minor confusion has been noted in third-party agentic testing where the model occasionally struggles with self-awareness in complex scaffolds, but its core identity remains stable and verifiable through official API and weight metadata.

The licensing is split: the code is under the permissive MIT license, but the model weights are governed by a custom 'DeepSeek Model License.' While this license explicitly allows commercial use and derivative works, it includes 'Use-based restrictions' and 'Accountability' clauses that are more restrictive than standard Open Source Initiative (OSI) licenses. The terms are public but create a more complex legal landscape than a standard Apache 2.0 or MIT license.

Hardware requirements are well-documented by both the provider and the community. The technical report discusses the use of FP8 mixed precision, and official guides specify VRAM requirements for various configurations (e.g., ~700GB for FP8 inference). Third-party documentation (e.g., vLLM, SGLang) provides detailed quantization trade-offs (INT4, GGUF) and multi-node requirements, though the provider's own documentation could be more centralized regarding consumer-grade hardware limits.

DeepSeek maintains a public changelog for its API and releases versioned weights (e.g., V3-0324, V3.1). However, the rapid release cycle and 'silent' updates to the hosted API (deepseek-chat) have led to reports of behavioral drift. While semantic versioning is partially used, the deprecation of older versions (like the original V3) happens quickly, sometimes leaving users with limited paths for long-term stability on a specific checkpoint.