Rule of Thumb: Parameters to VRAM

Estimating the hardware needed to run a Large Language Model (LLM) can seem complex. A straightforward approximation helps simplify this process. The most significant factor determining the memory requirement, specifically the Graphics Processing Unit's Video RAM (VRAM), is the size of the model itself, measured by its number of parameters.

Think of each parameter in the model as a number that needs to be stored somewhere accessible for computation. Since GPUs are the workhorses for running LLMs (as discussed in Chapter 2), these parameters are primarily loaded into the GPU's dedicated memory, the VRAM.

How much space does each parameter take? This depends on the precision or data type used to store it. Common precisions include:

FP32 (Single-Precision Floating Point): Each parameter takes 32 bits, which is equal to 4 bytes. This offers high precision but requires more memory.
FP16 (Half-Precision Floating Point): Each parameter takes 16 bits, or 2 bytes. This cuts the memory requirement roughly in half compared to FP32, often with a minimal impact on performance for inference.
INT8 (8-bit Integer): Each parameter takes only 8 bits, or 1 byte. This further reduces memory usage significantly but can sometimes lead to a noticeable decrease in the model's accuracy. This is often achieved through a process called quantization (which we briefly introduced in Chapter 3).

The most common starting point for estimation assumes the model will be run using FP16 precision, as it offers a good balance between memory usage and performance.

The Basic Calculation

Based on this, we can establish a simple rule of thumb to estimate the minimum VRAM needed just to load the model parameters:

$Required \, VRAM \, (GB) \approx \frac{Parameter \, Count \, (Billions) \times Bytes \, Per \, Parameter}{1024^3 \, (Bytes/GB)}$

However, a simpler mental shortcut, especially for FP16, is often used:

$Required \, VRAM \, (GB) \approx Parameter \, Count \, (Billions) \times 2 \, (Bytes \, for \, FP16)$

Let's look at a couple of examples:

A 7 Billion Parameter Model (e.g., Llama 2 7B):
- Using FP16 precision (2 bytes/parameter): $VRAM \approx 7 \, \text{Billion} \times 2 \, \text{Bytes} = 14 \, \text{Billion Bytes}$ $VRAM \approx \frac{14 \times 10^9}{1024^3} \approx 13.04 \, GB$ So, you'd need approximately 14 GB of VRAM just to hold the model weights in FP16.
- Using FP32 precision (4 bytes/parameter): $VRAM \approx 7 \, \text{Billion} \times 4 \, \text{Bytes} = 28 \, \text{Billion Bytes}$ $VRAM \approx \frac{28 \times 10^9}{1024^3} \approx 26.07 \, GB$ Loading the same model in full precision would require about 28 GB of VRAM.
- Using INT8 precision (1 byte/parameter, post-quantization): $VRAM \approx 7 \, \text{Billion} \times 1 \, \text{Byte} = 7 \, \text{Billion Bytes}$ $VRAM \approx \frac{7 \times 10^9}{1024^3} \approx 6.52 \, GB$ Using 8-bit quantization significantly reduces the requirement to roughly 7 GB of VRAM.
A 70 Billion Parameter Model (e.g., Llama 2 70B):
- Using FP16 precision (2 bytes/parameter): $VRAM \approx 70 \, \text{Billion} \times 2 \, \text{Bytes} = 140 \, \text{Billion Bytes}$ $VRAM \approx \frac{140 \times 10^9}{1024^3} \approx 130.36 \, GB$ This larger model demands around 140 GB of VRAM in FP16. This often requires multiple high-end GPUs.

Estimated VRAM needed solely for storing the parameters of a 7 Billion parameter model at different numerical precisions.

Important Notes

This rule of thumb provides a baseline estimate for storing the model's parameters. It's a fundamental starting point for determining if your hardware might be sufficient. However, remember that this calculation only accounts for the model weights themselves. Running the model involves more than just storing it. As we'll discuss next, factors like activations during inference, the length of the input prompt and generated output (context length), and software overhead will consume additional VRAM. Therefore, you should always budget for more VRAM than this simple calculation suggests.

Was this section helpful?

References

Automatic Mixed Precision for Deep Learning, NVIDIA Developer Documentation, 2023 (NVIDIA) - Explains the principles and benefits of using mixed precision (FP16 and FP32) in deep learning, crucial for understanding memory and performance optimizations on GPUs.
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Andrey Bochenin, Vitaly Tarasov, Andrew Karpov, Dianne Jouppi, Anujan Varma, Gabriel Micha, 2018 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE) DOI: 10.1109/CVPR.2018.00696 - A foundational academic paper introducing techniques for quantizing neural networks to 8-bit integers (INT8) for efficient inference, directly relevant to memory reduction.
VRAM estimation for large models, Hugging Face documentation contributors, 2023 (Hugging Face) - A practical guide from a leading LLM platform detailing how to estimate VRAM requirements for large language models, including considerations for different data types and additional memory overheads.