While pre-trained large language models (LLMs) demonstrate impressive capabilities in understanding and generating human language based on patterns learned from vast datasets, they don't automatically behave in ways that are consistently useful, truthful, or safe for specific applications. Their training objective, typically focused on predicting the next token in a sequence, $P(\text{token}_{i+1} | \text{token}_1, ..., \text{token}_i)$ , maximizes likelihood on the pre-training corpus but doesn't directly optimize for following user instructions or adhering to human values.

Alignment is the process of refining a pre-trained LLM to better match human intent and preferences. It aims to steer the model's powerful generative abilities towards desired behaviors. Supervised Fine-Tuning (SFT), the focus of this chapter, represents a significant first step in this process. The overarching goals of alignment, which SFT begins to address, are often categorized into three broad areas, sometimes referred to as the "HHH" criteria: Helpfulness, Honesty, and Harmlessness.

Helpfulness

This is perhaps the most direct goal addressed by SFT. A helpful model should understand and accurately follow the user's instructions as presented in the prompt. It should perform the requested task effectively, whether it's answering a question, summarizing text, writing code, translating languages, or engaging in a specific conversational style.

Consider a pre-trained model asked, "Explain the concept of gradient descent."

Unhelpful (but plausible pre-trained behavior): The model might generate text defining gradients in calculus generally, or list papers that mention gradient descent, without providing a clear explanation of the algorithm itself.
Helpful (aligned behavior): The model provides a clear, step-by-step explanation tailored to the likely context (e.g., machine learning), possibly including analogies or simple examples.

SFT achieves this by exposing the model to numerous examples of prompts paired with high-quality, helpful responses. The fine-tuning process adjusts the model's parameters to increase the probability of generating such helpful responses for similar prompts. This involves minimizing the loss (e.g., cross-entropy) between the model's generated response and the target helpful response in the SFT dataset.

Alignment transforms a general-purpose pre-trained model into one that exhibits desired behaviors through techniques like SFT.

Honesty (Truthfulness)

An aligned model should strive for accuracy and avoid generating fabricated information, often referred to as "hallucinations." While pre-training exposes the model to factual knowledge, the generative nature means it can easily construct plausible-sounding but incorrect statements. Honesty implies:

Factual Accuracy: Providing correct information when possible.
Avoiding Fabrication: Not inventing facts, sources, or details.
Expressing Uncertainty: Indicating when it does not know the answer or when information is ambiguous, rather than guessing confidently.

SFT can contribute to honesty by including examples where the model correctly answers factual questions or explicitly states its limitations. However, ensuring deep factuality and calibrated uncertainty often requires more advanced techniques beyond basic SFT, such as incorporating retrieval mechanisms or using reinforcement learning (like RLHF, discussed in Chapter 26) to penalize untruthful outputs identified by human feedback.

Harmlessness

This goal focuses on preventing the model from generating output that is harmful, unethical, prejudiced, toxic, or promotes illegal activities. Pre-training data inevitably contains biases and harmful content present on the internet and in digitized texts. An unaligned model might readily reproduce or amplify these issues. Harmlessness requires the model to:

Refuse Inappropriate Requests: Decline to generate content that falls into harmful categories.
Avoid Bias: Reduce the generation of stereotyped or prejudiced content.
Maintain Safety: Not provide instructions for dangerous activities.

SFT plays a role here by including examples where the model refuses harmful requests or provides safe, neutral responses. Carefully curated SFT datasets filter out undesirable examples and explicitly demonstrate safe refusals. Similar to honesty, achieving robust harmlessness across diverse and adversarial inputs is challenging and often benefits significantly from subsequent RLHF, where models are trained to prefer safe outputs based on human judgments.

In summary, alignment aims to make LLMs not just capable, but also beneficial and safe partners in various applications. SFT serves as a foundational technique, primarily enhancing helpfulness and instruction-following, while also starting the process of instilling honesty and harmlessness by providing concrete examples of desired model outputs. These goals guide the creation of SFT datasets and the evaluation of aligned models, ensuring they move beyond simply predicting text to generating truly useful and responsible responses.