Understanding LLM Model Sizes and Hardware Requirements
Chapter 1: Introduction to Large Language Models and Size
What is a Large Language Model (LLM)?
Understanding Model Parameters
How Model Size is Measured
Examples of Different Model Sizes
Quiz for Chapter 1
Chapter 2: Essential Hardware Components for AI
The Central Processing Unit (CPU)
Random Access Memory (RAM)
The Graphics Processing Unit (GPU)
Video RAM (VRAM)
Brief Overview of TPUs
Quiz for Chapter 2
Chapter 3: Connecting Model Size to Hardware Needs
Model Parameters and Memory Consumption
Data Types and Precision (FP16, INT8)
Introduction to Quantization
Compute Requirements (FLOPS)
Memory Bandwidth Importance
Quiz for Chapter 3
Chapter 4: Running LLMs: Inference vs. Training
What is Model Inference?
Hardware Needs for Inference
What is Model Training?
Hardware Needs for Training
Focus on Inference Requirements
Quiz for Chapter 4
Chapter 5: Estimating Hardware Needs
Rule of Thumb: Parameters to VRAM
Accounting for Activation Memory
Factors Influencing Actual Usage
Checking Hardware Specifications
Practice: Simple VRAM Estimations
Quiz for Chapter 5
Introduction to Quantization
Was this section helpful?
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022 NeurIPS 2022 DOI: 10.48550/arXiv.2208.07339 - Presents a widely adopted method for 8-bit quantization for large Transformer models, enabling them to run on consumer GPUs.
- GPTQ: Accurate Post-Training Quantization for Generative Pretrained Transformers, Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2023 International Conference on Learning Representations DOI: 10.48550/arXiv.2210.17323 - Introduces a technique for post-training quantization of large language models to 4-bit precision with minimal accuracy reduction.
- QLoRA: Efficient Finetuning of Quantized LLMs on Consumer GPUs, Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer, 2023 Advances in Neural Information Processing Systems (NeurIPS) DOI: 10.48550/arXiv.2305.14314 - Describes a method that uses 4-bit quantization for efficient finetuning of large language models, showcasing the practical utility of quantization.
- bitsandbytes Library Documentation, Tim Dettmers and contributors, 2024 (bitsandbytes-foundation) - Official documentation for the Python library providing 8-bit and 4-bit quantization and optimizers for PyTorch, widely used for memory-efficient LLM deployment.