Chapter 1 outlined the computational and memory challenges associated with fine-tuning large language models. This chapter addresses these issues by concentrating on Low-Rank Adaptation (LoRA), a specific and widely used Parameter-Efficient Fine-Tuning (PEFT) technique.

We start with the core hypothesis behind LoRA: the change in model weights during adaptation, $\Delta W$, has a low "intrinsic rank". This suggests that $\Delta W$ can be effectively approximated by the product of two much smaller matrices, $B$ and $A$.

$\Delta W \approx BA$

Where $W \in \mathbb{R}^{d \times k}$, $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and the rank $r \ll \min(d, k)$.

Throughout this chapter, you will learn:

• The theoretical motivation and mathematical formulation of LoRA.
• How LoRA decomposes weight updates into low-rank matrices ($A$ and $B$).
• Practical considerations for selecting the rank $r$ and configuring the scaling parameter $\alpha$.
• How to implement LoRA modifications within neural network layers, particularly linear layers.
• Methods for integrating LoRA into standard Transformer blocks, targeting specific components like attention or feed-forward networks.

By the end of this chapter, you will understand the mechanics of LoRA and be prepared to implement its basic form for efficient LLM fine-tuning.