Meta provides a comprehensive 92-page technical paper detailing the Llama 3.1 405B architecture. It is explicitly described as a dense, decoder-only Transformer. Key architectural choices are documented, including Grouped-Query Attention (GQA) with 8 KV heads, 126 layers, a hidden dimension of 16384, and the use of RoPE, RMSNorm, and SwiGLU. The training methodology, including the multi-stage post-training pipeline (SFT, RS, DPO), is thoroughly described. While the paper is extensive, some specific hyperparameter tuning details for the 405B variant remain proprietary.

Meta discloses that the model was trained on ~15 trillion tokens with a cutoff of December 2023. The paper provides a high-level breakdown of the data mix (e.g., 5% multilingual, emphasis on math and code) and describes the cleaning and filtering pipeline (using fastText classifiers and Llama-based quality scoring). However, it fails to name specific data sources beyond 'publicly available web data,' providing no list of domains or specific datasets used, which is a significant gap in transparency regarding data provenance.

The tokenizer is publicly available via the official GitHub repository and Hugging Face. It uses a Tiktoken-based BPE approach with a large vocabulary of 128,256 tokens, which is a significant increase from Llama 2 to better support multilingual text. The vocabulary and tokenization logic are fully inspectable and well-documented in the technical paper, including the handling of reserved tokens for special functions.

The model is explicitly stated to be a dense architecture with 405 billion parameters. Unlike MoE models, all parameters are active during inference, which is clearly documented. Detailed architectural specifications (layers, heads, dimensions) allow for precise verification of the parameter count. Some third-party sources (e.g., Fireworks AI) report slightly different counts (410B) likely due to embedding layer overhead, but Meta's official documentation is consistent.

Meta provides specific compute metrics: 39.3 million H100 GPU hours for the entire Llama 3.1 family, with the 405B model utilizing over 16,000 H100s. The paper details the hardware infrastructure (custom-built clusters), training duration (~54 days for the final run), and environmental impact (estimated 11,390 tons CO2eq, though Meta claims 0 tons market-based due to offsets). This level of compute transparency is significantly higher than most industry peers.

Meta released an 'eval_details.md' on GitHub and a collection of evaluation datasets on Hugging Face to aid reproduction. They specify few-shot settings and prompt formats for major benchmarks like MMLU and GSM8K. However, the evaluation code itself is not fully open-sourced in a 'push-button' format, and the use of internal evaluation libraries makes exact bit-for-bit reproduction difficult for independent researchers.

Llama 3.1 405B demonstrates high identity consistency, correctly identifying itself and its version in most standard deployments. It is transparent about its nature as an AI developed by Meta. The model card and system prompts are designed to maintain this identity, and there are no widespread reports of the model claiming to be a competitor's product.

The 'Llama 3.1 Community License Agreement' is publicly available and explicitly allows for commercial use and the use of outputs to train other models (a major transparency improvement). However, it is not a standard OSI-approved open-source license. It contains a '700 million monthly active users' restriction and requires 'Built with Llama' attribution, creating legal complexity that prevents a higher score.

Hardware requirements are well-documented by both Meta and the community. Official documentation specifies VRAM needs for FP16 (~810GB), FP8 (~405GB), and INT4 (~203GB). Meta also provides an official FP8 quantized version and documents the minimal accuracy trade-offs. The impact of the 128k context window on KV cache memory is also detailed, providing clear guidance for deployment.

Meta uses a versioning system (e.g., Llama 3.1), but it lacks strict semantic versioning for weight updates. There have been instances of 'silent' updates to the weights on Hugging Face (e.g., the August 2024 update to fix memory issues) without a clear, public-facing changelog or version history for previous weight iterations, making it difficult to track behavioral drift over time.