The Reinforcement Learning from Human Feedback (RLHF) process typically begins with Supervised Fine-Tuning (SFT). This initial stage adapts a general pre-trained Large Language Model (LLM) to better suit the target task or domain before the reinforcement learning phase. SFT uses a dataset of high-quality prompt-response examples to provide the model with a strong baseline understanding of the desired behavior and output format.

This chapter focuses on the SFT phase. You will learn about:

• The function of SFT in initializing the policy for RLHF.
• Methods for curating suitable demonstration datasets.
• Key implementation aspects, including training configurations and hyperparameters.
• Techniques for evaluating the performance of the SFT model.

We will conclude with a practical exercise demonstrating how to perform SFT on a language model. Understanding SFT is essential for building an effective RLHF pipeline.