Scaling Laws for Neural Language Models

When scaling Transformer models, a natural question arises: how do we best invest our computational resources? Should we prioritize adding more layers, widening the hidden dimensions, or training on significantly more data? Simply increasing parameters without a clear strategy can lead to diminishing returns or inefficient use of expensive compute cycles. Fortunately, empirical studies have revealed predictable relationships between model performance, model size, dataset size, and the amount of computation used for training. These relationships are often referred to as "scaling laws".

Understanding these scaling laws provides a valuable framework for making informed decisions about model architecture and training regimes. They allow us to estimate the expected performance gains from scaling up different aspects of the training process and help optimize the allocation of our computational budget.

The Empirical Basis of Scaling Laws

Pioneering work, notably by Kaplan et al. (2020) from OpenAI, demonstrated that the performance of language models, typically measured by the cross-entropy loss on a held-out test set, improves predictably as we scale up model size (number of non-embedding parameters, $N$), dataset size (number of training tokens, $D$), and compute (total floating-point operations, FLOPs, $C$).

The core finding is that the test loss $L$ often follows a power-law relationship with $N$, $D$, and $C$, when other factors are not the bottleneck. For model size $N$ and dataset size $D$, the relationship can often be modeled as:

$$L(N, D) \approx L_{\infty} + \left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D}$$

Here:

• $L(N, D)$ is the predicted cross-entropy loss.
• $L_{\infty}$ represents an irreducible loss floor, potentially related to the inherent entropy of language itself or the limits of the modeling approach.
• $N$ is the number of non-embedding parameters in the model.
• $D$ is the number of tokens in the training dataset.
• $N_c$, $D_c$, $\alpha_N$, and $\alpha_D$ are constants specific to the model architecture and training setup, determined empirically by fitting the formula to results from training runs of various smaller scales. Typically, $\alpha_N$ and $\alpha_D$ are less than 1, indicating diminishing returns as $N$ or $D$ increases.

This formula suggests that the loss is dominated by the term corresponding to the limiting factor: if the model is too small ($N \ll N_c$), increasing $N$ helps significantly; if the dataset is too small ($D \ll D_c$), increasing $D$ is more beneficial.

Similarly, the loss often scales as a power law with the compute budget $C$:

$$L(C) \approx L_{\infty} + \left(\frac{C_c}{C}\right)^{\alpha_C}$$

Where $C_c$ and $\alpha_C$ are again empirically determined constants. These relationships typically hold over several orders of magnitude, making them useful for extrapolation.

A log-log plot illustrating how test loss might decrease as model size increases, often following a predictable power law over several orders of magnitude.

Compute-Optimal Scaling and the Chinchilla Findings

A significant refinement to understanding scaling laws came from Hoffmann et al. (2022) at DeepMind, known as the "Chinchilla" study. They performed a careful analysis to determine the optimal allocation of a fixed compute budget $C$ between model size $N$ and data size $D$.

The approximate relationship between compute, model size, and data size for training dense transformer models is often estimated as:

$$C \approx 6 \times N \times D$$

This reflects that the computational cost is roughly proportional to the number of parameters multiplied by the number of tokens processed. Given a fixed compute budget $C$, the Chinchilla study suggested that for optimal performance (lowest loss), both $N$ and $D$ should be scaled roughly in proportion to the square root of the compute budget. That is, if you double the compute budget, you should aim to increase both model size and dataset size by a factor of about $\sqrt{2} \approx 1.4$.

This finding was important because it suggested that many large language models trained prior to this study (like GPT-3 or Gopher) might have been "over-parameterized" relative to their training data. For the compute budgets used, better performance might have been achieved with smaller models trained on significantly more data. The Chinchilla model itself, trained according to these "compute-optimal" principles, achieved state-of-the-art results with fewer parameters but more training data than some of its larger contemporaries.

Let's illustrate with a simplified calculation. Suppose we fit a scaling law and find optimal performance occurs when $N \approx k \sqrt{C}$ and $D \approx \frac{C}{6N} \approx \frac{C}{6k\sqrt{C}} = \frac{\sqrt{C}}{6k}$ for some constant $k$. If we have a compute budget $C_1$ and find the optimal $N_1, D_1$, then for a larger budget $C_2 = 4 C_1$, the optimal parameters would be approximately $N_2 \approx k \sqrt{4C_1} = 2 N_1$ and $D_2 \approx \frac{\sqrt{4C_1}}{6k} = 2 D_1$. We scale both model and data proportionally to $\sqrt{C}$.

Practical Implications for Training LLMs

These scaling laws have several direct consequences for engineers building large models:

1. Resource Allocation: They provide quantitative guidance on whether to invest in more compute hardware (to increase $N$ and $D$), more data acquisition and cleaning (to increase $D$), or architectural improvements that might change the constants ($N_c, D_c, \alpha_N, \alpha_D$).
2. Performance Prediction: By running smaller-scale experiments to estimate the scaling parameters ($\alpha_N, \alpha_D$, etc.), teams can predict the likely performance of a much larger model before committing the significant resources required for the full training run. This reduces risk and helps justify compute requests.
3. Budget Optimization: For a fixed budget $C$, the laws help determine the optimal balance between $N$ and $D$ to maximize expected performance. This prevents scenarios where a model is severely bottlenecked by either its size or the data it was trained on.
4. Benchmarking: Scaling laws provide a baseline for evaluating new architectures or training techniques. If a new method yields performance significantly better than predicted by the established scaling laws for standard Transformers, it suggests a genuine improvement.

Estimating Compute Requirements

We can use the $C \approx 6ND$ formula to estimate the computational cost of training. For instance, training a 7 billion parameter model ($N=7 \times 10^9$) on 1 trillion tokens ($D=1 \times 10^{12}$) would require approximately:

import math

# Approximate number of parameters (excluding embeddings, simplified)
N = 7e9
# Number of training tokens
D = 1e12

# Estimated FLOPs using the 6ND rule
C_flops = 6 * N * D

# Convert FLOPs to Petaflop-days (1 Petaflop = 1e15 FLOP/s)
# Seconds in a day = 86400
petaflops = 1e15
seconds_per_day = 86400

C_petaflop_days = C_flops / (petaflops * seconds_per_day)

print(f"Estimated Compute: {C_flops:.2e} FLOPs")
print(f"Estimated Compute: {C_petaflop_days:.2f} Petaflop-days")
# Output:
# Estimated Compute: 4.20e+22 FLOPs
# Estimated Compute: 486.11 Petaflop-days

This calculation highlights the immense computational scale involved. A cluster capable of sustained 10 Petaflops would take roughly 49 days for such a run, ignoring overheads and potential inefficiencies. This underscores why efficient resource allocation guided by scaling laws is so important.

Caveats

While powerful, it's necessary to remember the context of these scaling laws:

• Empirical Nature: They are observed trends, not universal laws. They depend heavily on the specific architecture (e.g., Transformer), optimizer, data distribution, and other training details. Extrapolating too far from the range of experiments used to derive the laws carries risk.
• Data Quality: The laws primarily model the quantity ($D$) of data. However, data quality, diversity, and relevance are also immensely significant factors influencing final model performance, which are not explicitly captured in these simple formulas. Training on extensive amounts of low-quality data may yield worse results than training on less, but cleaner, data.
• Irreducible Loss: The $L_{\infty}$ term suggests a performance limit. As models get extremely large and are trained on extensive datasets, improvements may slow down as they approach this limit.
• Compute Estimation: The $6ND$ formula is a rough approximation. Actual FLOPs can vary depending on the exact implementation, sequence length, and hardware efficiency. More precise calculations might consider forward and backward passes separately.

In summary, scaling laws provide an invaluable quantitative lens through which to view the process of building larger and more capable language models. They transform scaling from a guessing game into a more predictable engineering discipline, enabling more efficient use of computational resources and providing a framework for assessing progress in the field. As we explore specific architectural choices in the following sections, keep these scaling principles in mind as they often motivate the design decisions made for large-scale Transformers.