Llama 3.3 70B

Llama 3.3 70B is explicitly documented as an auto-regressive dense Transformer model. Meta provides detailed technical specifications including the use of Grouped-Query Attention (GQA) for inference efficiency and a 128k token context window. The model's evolution from Llama 3.1 is clear, utilizing similar architectural foundations but with updated post-training methodologies (SFT and RLHF). While the high-level architecture is well-documented in the Llama 3 technical report and model cards, specific low-level architectural modifications unique to the 3.3 variant are described more as 'optimizations' rather than fully detailed structural changes.

Meta discloses that the model was pretrained on approximately 15 trillion tokens from 'publicly available online sources' with a cutoff of December 2023. For fine-tuning, they mention using over 25 million synthetic examples and publicly available instruction datasets. However, there is no specific breakdown of the data sources (e.g., percentage of code, web, books) or detailed disclosure of the filtering and cleaning methodologies beyond general mentions of heuristic and NSFW filters. The lack of granular composition data remains a significant transparency gap.

The tokenizer is publicly accessible via the official GitHub repository and Hugging Face. It features a vocabulary size of 128,256 tokens, which is a significant increase from the 32k used in Llama 2, aimed at improving multilingual efficiency. The tokenization approach is well-documented, and the vocabulary is consistent across official API and local implementations. The alignment with the claimed 8 supported languages is verifiable through the tokenizer's performance on those scripts.

The model is clearly defined as a dense architecture with 70.6 billion total parameters. Unlike Mixture-of-Experts (MoE) models where active parameters can be obscured, Llama 3.3 70B's dense nature means all parameters are active during inference. The parameter count is consistent across all official documentation, and the architectural breakdown (e.g., GQA implementation) is provided in technical reports.

Meta provides specific compute metrics, stating that training utilized approximately 39.3 million GPU hours on H100-80GB hardware (700W TDP). They also disclose the estimated environmental impact, citing 11,390 tons of CO2eq for the training process. While the hardware type and total hours are clear, the specific cluster configuration and exact training duration in days/months are less explicitly detailed compared to the most transparent research papers.

Meta publishes scores for standard benchmarks like MMLU, GPQA, and HumanEval. However, the exact prompts and few-shot examples used to achieve these specific scores are not always fully disclosed in a single reproducible repository. While third-party tools like 'lm_eval' can be used to approximate these results, discrepancies between official claims and independent audits are common, and the lack of a 'one-click' reproduction script for official numbers limits transparency.

The model consistently identifies itself as a Meta Llama model and is aware of its versioning (Llama 3.3). It maintains a coherent identity across different platforms and does not typically claim to be a competitor's model. Its capabilities and limitations, such as being text-only and having a December 2023 knowledge cutoff, are clearly stated in the model card and reflected in its behavior.

The model uses the 'Llama 3.3 Community License,' which is a custom license rather than a standard OSI-approved open-source license like Apache 2.0. While it allows for commercial use and derivative works, it includes a significant restriction: companies with over 700 million monthly active users must request a separate license from Meta. This 'open weights' but not 'open source' distinction is clearly stated but introduces legal complexity for large-scale users.

VRAM requirements are well-documented by both Meta and the community. For the 70B model, approximately 140GB of VRAM is required for FP16, while 4-bit quantization (INT4) reduces this to roughly 35-40GB. Meta provides guidance on using tools like bitsandbytes for quantization. However, official documentation on the specific accuracy-performance tradeoffs for various quantization levels (e.g., PPL loss per bit) is less comprehensive than community-driven benchmarks.

Meta uses a versioning system (3.1, 3.2, 3.3), but the changelogs are often high-level, focusing on 'improved reasoning' or 'better coding' rather than detailed technical diffs of weight changes or specific safety alignment shifts. There is no formal mechanism provided by Meta to access specific 'sub-versions' if weights are updated silently, and documentation on model drift over time is primarily left to third-party researchers.