Llama 3 8B

Meta provides comprehensive documentation for the Llama 3 architecture in their official technical report and model cards. The 8B variant is explicitly defined as a dense, decoder-only transformer with 32 layers, a hidden dimension of 4096, and 32 attention heads. Key technical modifications like Grouped-Query Attention (GQA) with 8 KV heads, SwiGLU activation, and Rotary Positional Embeddings (RoPE) with a base frequency of 500,000 are clearly documented. The training methodology, including the use of RMSNorm for pre-normalization and the specific AdamW optimizer hyperparameters, is publicly available.

While Meta discloses the scale of the pre-training data (15T+ tokens) and provides a high-level categorical breakdown (e.g., 5% non-English across 30+ languages, with specific mentions of code and mathematics), they do not release the actual dataset or provide a granular percentage-based composition of sources. Documentation mentions 'publicly available online data' and describes filtering/cleaning steps (PII removal, deduplication, and quality filtering), but the lack of specific source naming or a detailed data mixture prevents a higher score.

The tokenizer is publicly accessible via the official GitHub repository and Hugging Face. It uses a TikToken-based BPE approach with a clearly stated vocabulary size of 128,256 tokens. This expanded vocabulary is documented as a key improvement for encoding efficiency across diverse domains. The tokenizer's behavior, including special tokens like <|begin_of_text|> and <|eot_id|>, is well-documented for both pre-trained and instruction-tuned variants, allowing for full verification and local testing.

The model is explicitly identified as a dense architecture with 8.03 billion total parameters. Meta provides a detailed breakdown of parameter allocation, such as the 1.05 billion parameters dedicated to the embedding and language modeling heads (12.5% of the total). There is no ambiguity regarding active vs. total parameters as it is not a Mixture-of-Experts (MoE) model. The impact of the large vocabulary on parameter density is clearly explained in technical documentation.

Meta provides specific details regarding the compute resources used for Llama 3 8B. Pre-training utilized approximately 1.3 million GPU hours on NVIDIA H100-80GB hardware. The report includes estimated carbon emissions (390 tCO2eq) and power consumption metrics. While the exact cost is not stated, the hardware specifications and duration allow for accurate third-party estimation. The use of custom training libraries and the Research SuperCluster (RSC) infrastructure is also documented.

Meta reports scores across standard benchmarks (MMLU, GSM8K, HumanEval) and provides some evaluation details, such as the number of shots and prompt styles (e.g., 8-shot CoT for GSM8K). However, the full evaluation code and exact prompt templates were not initially released in a centralized, easily reproducible format. Independent researchers have noted difficulties in matching reported scores exactly due to subtle differences in prompting and parsing, though Meta has since released some 'eval_details' on GitHub to mitigate this.

The instruction-tuned variant of Llama 3 8B demonstrates high identity consistency, correctly identifying itself as a model trained by Meta. It maintains a clear versioning identity (Llama 3 vs 3.1) and is transparent about its status as an AI. The model's self-recognition capabilities are documented as an emergent behavior reinforced during the RLHF and alignment phases.

The model is released under the 'Meta Llama 3 Community License Agreement.' While the license is public and allows for commercial use, it is not a standard OSI-approved open-source license. It contains significant restrictions, including a requirement for a separate license if the user has more than 700 million monthly active users and a non-compete clause regarding the use of Llama to improve other models. These custom terms create legal complexity compared to Apache 2.0 or MIT licenses.

VRAM requirements are well-documented by both Meta and the community. For the standard BF16 precision, the model requires approximately 15-16GB of VRAM, while 4-bit quantized versions (Q4_K_M) are documented to run on ~5-6GB. Meta provides guidance on context length memory scaling (8k default) and the impact of GQA on inference efficiency. Quantization tradeoffs are widely discussed in community documentation and official model cards.

Meta uses a versioning system (Llama 3, 3.1, 3.2) and maintains a basic changelog on GitHub. However, the transition from Llama 3 to 3.1 involved significant changes in capabilities (e.g., context window expansion to 128k) that were not always clearly communicated as distinct from the base 8B model in early marketing. There have been reports of behavioral drift in instruction-following performance across minor weight updates, with limited public documentation on the specific delta between these iterations.