Fine-tuning large language models (LLMs) is a standard technique for adapting them to specific tasks. However, modifying all parameters of models containing billions of parameters, often represented as a weight matrix $W$, incurs substantial computational and memory costs. This chapter examines these challenges inherent in full fine-tuning.

You will learn about:

• The specific memory, compute, and storage requirements of full fine-tuning.
• The factors making parameter-efficient alternatives necessary.
• The mathematical background of Singular Value Decomposition (SVD), relevant for understanding methods like LoRA, often used for low-rank approximations like $M \approx U \Sigma V^T$.
• A classification of different Parameter-Efficient Fine-Tuning (PEFT) approaches.

This foundational knowledge will illustrate why PEFT methods have become essential and introduce the categories of techniques explored throughout this course.