Understanding LLM Model Sizes and Hardware Requirements
Chapter 1: Introduction to Large Language Models and Size
What is a Large Language Model (LLM)?
Understanding Model Parameters
How Model Size is Measured
Examples of Different Model Sizes
Quiz for Chapter 1
Chapter 2: Essential Hardware Components for AI
The Central Processing Unit (CPU)
Random Access Memory (RAM)
The Graphics Processing Unit (GPU)
Video RAM (VRAM)
Brief Overview of TPUs
Quiz for Chapter 2
Chapter 3: Connecting Model Size to Hardware Needs
Model Parameters and Memory Consumption
Data Types and Precision (FP16, INT8)
Introduction to Quantization
Compute Requirements (FLOPS)
Memory Bandwidth Importance
Quiz for Chapter 3
Chapter 4: Running LLMs: Inference vs. Training
What is Model Inference?
Hardware Needs for Inference
What is Model Training?
Hardware Needs for Training
Focus on Inference Requirements
Quiz for Chapter 4
Chapter 5: Estimating Hardware Needs
Rule of Thumb: Parameters to VRAM
Accounting for Activation Memory
Factors Influencing Actual Usage
Checking Hardware Specifications
Practice: Simple VRAM Estimations
Quiz for Chapter 5
Data Types and Precision (FP16, INT8)
Was this section helpful?
- Mixed-Precision Training, Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, Hao Wu, 2018 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.1710.03740 - Explains the methods of using FP16 and FP32 together for training deep neural networks, reducing memory and increasing speed.
- Precision for Deep Learning: From FP32 to INT8, Alexey Shcherbakov, 2021 (NVIDIA Developer Blog) - Provides a practical overview of different numerical precisions used in deep learning and their impact on performance and memory on NVIDIA GPUs.
- Deep Learning (Chapter 4: Numerical Computation), Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - Provides foundational knowledge on numerical computation, including floating-point arithmetic and its considerations in deep learning.