How Model Size is Measured

Large Language Models (LLMs) learn adjustable values called parameters during their training. A central question is how to quantify the 'size' of these massive models.

The most common and standard way to measure the size of a Large Language Model is by counting its total number of trainable parameters. Think of these parameters, as discussed previously, like the internal knobs and dials the model adjusts to learn patterns and relationships in the language data it's trained on.

Why use the parameter count? It serves as a direct indicator of the model's potential complexity and capacity. A model with more parameters generally has the potential to store more information, capture finer nuances in language, and potentially perform better on complex tasks compared to a model with fewer parameters. It's a useful, though not perfect, proxy for how capable a model might be.

You'll often hear LLM sizes discussed using a specific unit: billions of parameters. This is frequently abbreviated with a ' $B$ '. For example:

A "7B model" means a model with approximately 7 billion ( $7 \times 10^9$ ) parameters.
A "70B model" has around 70 billion parameters.
Some research models reach hundreds of billions, like 175B, or even cross into the trillions ( $T$ ) of parameters.

The sheer scale is what puts the "Large" in Large Language Models. These numbers are significantly bigger than those found in many earlier types of machine learning models.

Approximate parameter counts for different LLM size categories. The logarithmic scale on the vertical axis helps visualize the significant differences in magnitude between these sizes.

This measurement, the parameter count, is important for this course because it directly impacts the computational resources needed. As we'll see, a larger number of parameters generally translates to higher requirements for memory (like VRAM in GPUs) and processing power. Knowing the parameter count is the first step in estimating the hardware you might need.

Was this section helpful?

References

Language Models are Few-Shot Learners, Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei, 2020 Advances in Neural Information Processing Systems (NeurIPS) DOI: 10.48550/arXiv.2005.14165 - Introduces GPT-3, a 175-billion-parameter language model, demonstrating the capabilities of models with significantly increased scale. This work established new benchmarks for what 'large' meant in LLMs and showed the impact of parameter count on few-shot learning.
Scaling Laws for Neural Language Models, Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, 2020 arXiv preprint arXiv:2001.08361 DOI: 10.48550/arXiv.2001.08361 - This paper systematically investigates how neural language model performance scales with model size (parameters), dataset size, and training compute. It provides theoretical and empirical insights into the relationship between parameter count and model capacity, performance, and resource requirements.