Yi-34B

The model is documented as a modified decoder-only Transformer. While the technical report claims it was 'trained from scratch,' it acknowledges using the Llama architecture as a base for its implementation. Specific modifications like Grouped-Query Attention (GQA), SwiGLU activation, and Rotary Position Embedding (RoPE) are disclosed. However, the initial release faced significant criticism for renaming Llama's internal tensor names without attribution, which was later corrected to improve compatibility. The 'from scratch' claim is partially undermined by the heavy reliance on Llama's structural design and code logic.

01.AI discloses that the model was trained on a 3.1 trillion token bilingual (English/Chinese) corpus. While they mention a 'rigorous pipeline' involving heuristic and learned filters, they provide no specific breakdown of data sources (e.g., percentages of web, code, or books). The methodology for data cleaning is described in general terms in the technical report, but the lack of source-level transparency or sample data availability limits verification.

The tokenizer is publicly available via the SentencePiece framework using Byte-Pair Encoding (BPE). The vocabulary size is explicitly stated as 64,000 tokens. Documentation details specific handling of numeric data (splitting into digits) and rare characters (unicode-byte fallback). The tokenizer is accessible for inspection in the official Hugging Face and GitHub repositories, allowing for direct verification of its alignment with the claimed bilingual support.

The model's total parameter count is clearly stated as 34.4 billion. As a dense model, all parameters are active during inference. Detailed architectural specifications are provided, including 60 layers and a hidden size of 7168. While the breakdown between attention and FFN parameters isn't explicitly tabulated, the structural constants are sufficient for independent calculation.

Information regarding training compute is extremely limited. While the technical report mentions the use of 'robust training infrastructure' and overtraining beyond Chinchilla optimality to 3.1T tokens, it fails to disclose the specific hardware (e.g., number of H100/A100 GPUs), total GPU hours, or the environmental impact/carbon footprint. This lack of detail makes the training cost and resource intensity unverifiable.

The technical report lists performance on standard benchmarks like MMLU, C-Eval, and GSM8K. However, it lacks comprehensive reproduction instructions, exact evaluation prompts, or public evaluation code. Third-party audits have raised concerns about the 'suspiciously high' MMLU scores compared to real-world performance, and independent researchers have noted potential data leakage issues in benchmarks like GSM8K, which 01.AI has not fully addressed with public decontamination logs.

The model generally identifies as an AI developed by 01.AI. However, early versions exhibited identity confusion due to the inherited Llama architecture and naming conventions, leading to instances where it was perceived as a Llama derivative rather than an independent model. While versioning (e.g., Yi-1.5) has improved this, the initial lack of clear identity boundaries and the 'oversight' in tensor naming significantly impacted its consistency score.

The model weights are released under the 'Yi Series Models Community License Agreement,' which is a custom license. While it allows for free commercial use, it requires an explicit application for companies with more than 200 million monthly active users. This 'open weights' but not 'open source' (per OSI definitions) approach creates some ambiguity for commercial users, although the terms are generally better documented than proprietary models.

01.AI provides excellent documentation for hardware requirements. They explicitly list VRAM needs for different batch sizes (e.g., 16GB for 4-bit quantization) and provide guidance for running on consumer-grade hardware like the RTX 4090. Quantization impact is documented, and they offer official 4-bit (AWQ) and 8-bit (GPTQ) versions to facilitate deployment.

The model family has seen updates (e.g., the transition to Yi-1.5 and the 200K context variants), but a formal, detailed changelog or semantic versioning system is not consistently maintained across all repositories. Users have reported behavioral drift and repetition issues in newer fine-tunes without clear documentation from the provider on what changed in the underlying weights or training mixture.