Having established the principles and implementation of Low-Rank Adaptation (LoRA), we now broaden our perspective to other significant Parameter-Efficient Fine-Tuning (PEFT) techniques. This chapter provides a comparative survey of alternative methods developed to address the computational demands of fine-tuning Large Language Models (LLMs).

You will learn about:

• Adapter Tuning: Understand the concept of inserting small, trainable adapter modules within the layers of a pre-trained model. We will cover their architecture and implementation details.
• Prefix Tuning: Examine how adding tunable, continuous vector prefixes to the input or hidden states can condition model behavior without modifying core weights.
• Prompt Tuning: Study techniques that focus on learning soft prompt embeddings, offering another way to guide model output efficiently.

Throughout this survey, we will analyze the architectural differences, the number of trainable parameters introduced, and the typical performance characteristics of each method. We will also compare their memory usage and computational requirements during training and inference, providing context for choosing the appropriate technique for specific constraints and objectives. The chapter concludes with a practical exercise implementing Adapter Tuning.