Instruction Following and Fine-tuning Review

Large Language Model (LLM) alignment addresses challenges in steering LLM behavior. This involves foundational techniques such as instruction following and supervised fine-tuning (SFT). These initial steps are primary to understanding how LLMs are controlled, forming a basis for advanced methods.

Raw, pre-trained LLMs are typically optimized for next-token prediction based on massive, unstructured text corpora. Their objective is often maximizing the likelihood of the training data under the model, typically via minimizing a cross-entropy loss:

$$L_{\text{pretrain}}(\theta) = - \sum_{i} \log P(x_i | x_{<i}; \theta)$$

where $\theta$ represents the model parameters and $x_i$ are tokens in the pre-training corpus. This process results in models with broad linguistic knowledge and generative capabilities but without specific instruction-following abilities or inherent adherence to safety guidelines. They learn grammar, facts, and reasoning patterns from the data but lack a specific goal beyond plausible text completion.

Instruction Fine-tuning (IFT)

To make pre-trained models more useful and controllable, a standard practice is Instruction Fine-tuning (IFT). This is a supervised learning phase where the model is further trained on a dataset composed of instructional prompts and desired responses.

The dataset $D_{\text{IFT}}$ takes the form of structured examples:

$$D_{\text{IFT}} = \{ (\text{prompt}_k, \text{completion}_k) \}_{k=1}^N$$

Examples might include:

• (prompt: "Translate the following sentence to Spanish: 'The weather is nice today.'", completion: "El clima está agradable hoy.")
• (prompt: "Summarize the main points of this paragraph: [Paragraph Text]", completion: "[Concise Summary]")
• (prompt: "Write Python code to reverse a string.", completion: def reverse_string(s):\n return s[::-1])

The optimization objective during IFT is to adjust the model parameters $\theta$ (starting from $\theta_{\text{pretrain}}$) to minimize the negative log-likelihood of generating the target completion tokens, given the prompt:

$$L_{\text{IFT}}(\theta) = - \sum_{k=1}^N \sum_{j=1}^{|c_k|} \log P(c_{k,j} | \text{prompt}_k, c_{k,<j}; \theta)$$

Here, $c_k = (c_{k,1}, ..., c_{k, |c_k|})$ is the token sequence of the desired completion_k. Essentially, the model learns: "When you see input like $\text{prompt}_k$, produce output like $\text{completion}_k$."

Basic workflow of Instruction Fine-tuning (IFT), adapting a pre-trained model using prompt-completion pairs.

IFT teaches the model the format of interaction (understanding instructions and providing relevant answers) and imbues it with specific capabilities reflected in the fine-tuning data.

Supervised Fine-tuning (SFT) in General

IFT is a specific type of Supervised Fine-tuning (SFT). More generally, SFT involves adapting a pre-trained model using any dataset of input-output pairs $(x, y)$, minimizing a loss function (like cross-entropy) computed on the target output $y$. In addition to instruction following, SFT can be used for:

• Domain Adaptation: Fine-tuning on specialized corpora (e.g., medical literature, legal documents) to improve performance and terminology usage within that domain.
• Style Adaptation: Fine-tuning on text with a specific style (e.g., formal, conversational) to guide the model's output tone.
• Simple Behavior Modification: Training on examples demonstrating desired behaviors, like refusing certain types of requests (though this is often handled better by more advanced alignment techniques).

Role in Alignment and Limitations

IFT and SFT are fundamental steps towards achieving outer alignment. They directly shape the model's observable behavior by optimizing it to mimic the provided examples. If the fine-tuning dataset consists of helpful, honest, and harmless examples, the model learns to produce similar outputs.

However, relying solely on SFT/IFT for alignment has significant limitations, motivating the advanced techniques discussed later:

1. Data Quality and Coverage: The alignment quality is entirely dependent on the SFT dataset. Creating a dataset that comprehensively covers all desired behaviors, nuances, and safety constraints is extremely challenging and costly. Any gaps, biases, or inconsistencies in the data will likely be learned by the model.
2. Scalability of Data Creation: Generating high-quality, diverse instruction-following or behavioral data requires significant human effort and expertise.
3. Handling Details and Preferences: SFT struggles with objectives that are difficult to capture in a single "correct" completion. Evaluating helpfulness, harmlessness, creativity, or adherence to complex ethical principles often involves comparing multiple possible outputs, which is not the standard SFT setup.
4. Specification Gaming: The SFT loss ($L_{\text{IFT}}$) acts as a proxy ($R_{\text{proxy}}$) for the true intended goal ($R_{\text{intended}}$), such as "be generally helpful and harmless". The model might find ways to minimize $L_{\text{IFT}}$ without truly satisfying $R_{\text{intended}}$. For example, it might learn overly cautious refusals that minimize the risk of generating bad content according to the SFT data, but also reduce helpfulness. It might generate outputs that superficially match the desired format but are factually incorrect or subtly misleading if those patterns weren't sufficiently penalized in the fine-tuning data.

In summary, IFT and SFT are powerful tools for making LLMs follow instructions and adopt specific knowledge or styles. They form the foundation upon which more advanced alignment techniques like RLHF are often built. However, their limitations in handling preferences, ensuring reliability, and preventing specification gaming necessitate the methods discussed throughout this course. They primarily address what the model should output based on examples, rather than optimizing directly for the underlying principles of desired behavior.