Understanding LLM Model Sizes and Hardware Requirements
Chapter 1: Introduction to Large Language Models and Size
What is a Large Language Model (LLM)?
Understanding Model Parameters
How Model Size is Measured
Examples of Different Model Sizes
Quiz for Chapter 1
Chapter 2: Essential Hardware Components for AI
The Central Processing Unit (CPU)
Random Access Memory (RAM)
The Graphics Processing Unit (GPU)
Video RAM (VRAM)
Brief Overview of TPUs
Quiz for Chapter 2
Chapter 3: Connecting Model Size to Hardware Needs
Model Parameters and Memory Consumption
Data Types and Precision (FP16, INT8)
Introduction to Quantization
Compute Requirements (FLOPS)
Memory Bandwidth Importance
Quiz for Chapter 3
Chapter 4: Running LLMs: Inference vs. Training
What is Model Inference?
Hardware Needs for Inference
What is Model Training?
Hardware Needs for Training
Focus on Inference Requirements
Quiz for Chapter 4
Chapter 5: Estimating Hardware Needs
Rule of Thumb: Parameters to VRAM
Accounting for Activation Memory
Factors Influencing Actual Usage
Checking Hardware Specifications
Practice: Simple VRAM Estimations
Quiz for Chapter 5
Accounting for Activation Memory
Was this section helpful?
- Attention Is All You Need, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, 2017 Advances in Neural Information Processing Systems 30 (NIPS 2017) DOI: 10.48550/arXiv.1706.03762 - Fundamental paper introducing the Transformer architecture, which forms the basis for modern LLMs and their activation generation.
- Deep Learning Systems: Algorithms, Compilers, and Processors for Large-Scale AI, Dawei Chen, Haibin Li, Zidong Zhang, 2023 (Springer) DOI: 10.1007/978-981-99-4704-5 - Provides a comprehensive overview of deep learning systems, including detailed discussions on memory management, hardware considerations, and optimization techniques for large models, pertinent to understanding VRAM usage.
- Hardware Requirements and Memory Considerations for Transformers, Hugging Face, 2024 - Official documentation providing practical advice on memory considerations and optimization for running Transformer models, including aspects like batch size and hardware requirements.
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022 Advances in Neural Information Processing Systems 35 (NeurIPS 2022) DOI: 10.48550/arXiv.2205.14135 - Introduces an optimized attention algorithm that significantly reduces HBM (GPU memory) reads/writes and memory footprint of activations, demonstrating the impact of architecture on VRAM.