Llama 3.1 8B

Meta provides a highly detailed technical paper ('The Llama 3 Herd of Models') that explicitly documents the architecture. It is a dense, decoder-only transformer with specific modifications like Grouped-Query Attention (GQA), Rotary Positional Embeddings (RoPE), and RMSNorm. The paper details the pretraining and post-training (SFT/RLHF) methodologies extensively, including the use of 15.6 trillion tokens and the transition from Llama 3 to 3.1 with expanded context support.

While Meta discloses the scale (15T+ tokens) and general categories (web, code, multilingual), they do not provide a granular breakdown of the dataset composition (e.g., specific percentages for each source). The documentation mentions 'publicly available sources' and 'carefully curated' data but lacks a public list of specific datasets or a sample for verification, citing competitive and safety reasons.

The tokenizer is publicly available via the official GitHub and Hugging Face repositories. It uses a Tiktoken-based BPE approach with a clearly stated vocabulary size of 128,256 tokens. Documentation details the improved efficiency for multilingual text compared to Llama 2, and the tokenizer files are accessible for independent verification and local testing.

The parameter count is precisely stated as 8.03 billion. As a dense model, all parameters are active during inference, which is explicitly confirmed in the technical documentation. The architectural breakdown (layers, hidden dimension, attention heads) is fully disclosed in the model card and technical paper.

Meta provides specific compute metrics, stating that the 8B model required approximately 1.46 million H100 GPU hours. They also disclose the hardware used (H100-80GB), the total energy consumption, and the estimated carbon footprint (420 tons CO2eq for the 8B variant). This level of detail is exemplary compared to industry peers.

Meta provides a dedicated 'eval_details.md' on GitHub and a Hugging Face collection with evaluation data. However, independent researchers have noted difficulties in exactly matching reported scores due to specific prompt formatting and few-shot configurations that are not always perfectly mirrored in standard evaluation harnesses. A 2-point penalty was applied for documented instances of benchmark contamination in multilingual datasets (e.g., XNLI, XQuAD) as reported in third-party audits.

The model consistently identifies itself as Llama 3.1 and is transparent about its versioning. It does not exhibit the identity confusion common in smaller fine-tuned models. It clearly states its capabilities and limitations in the model card, and its behavior aligns with its documented 128k context window and multilingual support.

The 'Llama 3.1 Community License' is publicly available but is not a standard Open Source license (OSI-compliant). It contains significant commercial restrictions, specifically requiring a separate license for entities with over 700 million monthly active users. A 3-point penalty was applied due to the 'Acceptable Use Policy' containing broad, subjective restrictions that can override the stated license terms and create legal ambiguity for developers.

VRAM requirements are well-documented by both Meta and the community for various precisions (FP16, INT8, 4-bit). The model card provides guidance on hardware compatibility (NVIDIA Ampere/Hopper), and third-party benchmarks (e.g., SaladCloud, vLLM) provide detailed throughput and memory scaling data for the 128k context window.

Meta uses clear semantic-like versioning (3.1) and maintains a changelog on GitHub. While they have been transparent about the transition from 3.0 to 3.1, there is less documentation regarding minor 'silent' updates to the safety filters or alignment layers that can occur between major releases, though weights are generally pinned to specific commit hashes on Hugging Face.