SEA-LION-7B

SEA-LION-7B is explicitly documented as being built on the MosaicML Pretrained Transformer (MPT) architecture. AI Singapore provides clear technical specifications including 32 layers, a hidden dimension of 4096, and 32 attention heads. They disclose a significant architectural modification: the use of absolute learned positional embeddings instead of the standard ALiBi used in MPT, which was chosen to provide a stable foundation for its 2,048-token context window. The model was trained from scratch, and the training methodology using MosaicML Composer is publicly stated.

The training data composition is disclosed with a high degree of granularity. The 980B token corpus is broken down by source and language: English (58.2% via RefinedWeb), Chinese (9.3% via mC4), and specific percentages for Southeast Asian languages (e.g., Indonesian 1.5%, Vietnamese 6.46%, Malay 0.29%). The 'SEA-LION-PILE' dataset is publicly named, and the methodology for filtering and cleaning (including the use of fasttext for language classification) is documented. However, access to the full raw dataset is restricted, preventing complete independent verification.

The model uses a custom SEABPETokenizer with a 256,000-token vocabulary, which is significantly larger than standard Western models to better represent Southeast Asian scripts. The tokenizer is publicly available on Hugging Face, and its training methodology (sampling 20M lines from the training data using SentencePiece) is clearly documented. The efficiency gains (token-to-word ratio) for regional languages are explicitly claimed and verifiable through the provided tokenizer files.

The model is a dense decoder-only transformer with 7.1 billion parameters. Detailed architectural parameters are provided in the model card and technical documentation, including layer counts, head dimensions, and vocabulary size. There is no ambiguity regarding active vs. total parameters as it is not an MoE architecture. The impact of quantization is also addressed through the official release of GGUF versions (Q2_K to Q8_0), providing transparency on density vs. performance trade-offs.

AI Singapore discloses the hardware used (32 instances of NVIDIA A100 40GB GPUs) and the training duration (22 days). While this allows for a rough estimate of compute resources, they do not provide an official carbon footprint calculation or a detailed energy consumption report. The hardware specifications are clear, but the environmental impact data is missing, which is a key requirement for a high score in this pillar.

The model was evaluated using the BHASA benchmark and the SEA-HELM framework, both of which have associated papers and GitHub repositories. Evaluation results for standard benchmarks (ARC, MMLU, HellaSwag) are provided. However, while the benchmarks are named and some methodology is described (e.g., zero-shot Indonesian prompts), the exact evaluation code and full prompt sets used for the internal BHASA tests are not as easily accessible as the model weights themselves, creating a moderate gap in full reproducibility.

SEA-LION-7B demonstrates high identity consistency. It is clearly branded as a Southeast Asian-centric model and does not attempt to mimic competitors like GPT-4 or Llama. The model card and technical reports maintain a consistent narrative regarding its purpose, versioning (v1), and limitations (e.g., 2048 context window). There are no documented cases of the model misrepresenting its origin or capabilities.

The base model is released under the MIT license, which is a highly permissive and clear open-source license. The instruction-tuned 'Research' variant uses a more restrictive CC BY-NC-SA 4.0 license, but this distinction is clearly labeled on the respective Hugging Face repositories. The terms for commercial use are explicitly stated for the base model, and there is no evidence of conflicting terms between the code and the weights.

VRAM requirements are well-documented for various precisions. The documentation notes that FP16 requires ~14-15GB, while 4-bit quantization (GGUF) fits within ~4.5GB. The availability of multiple quantized versions (Q2 through Q8) on the official repository provides users with clear paths for deployment on consumer hardware. However, detailed memory scaling charts for context length vs. batch size are not explicitly provided in the core documentation.

While the model uses a versioning system (v1, v2, v3), the documentation for 'v1' specifically lacks a detailed, granular changelog for minor updates or weight refreshes. The transition between versions is documented at a high level (e.g., moving from MPT to Llama/Gemma bases in later versions), but the tracking of silent drift or performance changes within the v1 lifecycle is minimal. This makes it difficult for developers to track specific behavioral changes over time.