Having established a working definition of alignment, let's formalize the alignment problem. At its core, the alignment problem is the challenge of ensuring that an AI system, specifically an LLM in our context, reliably behaves in accordance with the designers' or users' intentions across a wide range of situations. This isn't merely about achieving high performance on a specific benchmark; it's about embodying desired principles like helpfulness, honesty, and harmlessness consistently.

Formalizing Alignment Objectives

Ideally, we want the LLM's behavior, represented by its conditional probability distribution $p_{\theta}(y|x)$ for generating output $y$ given input $x$ , to match some target distribution $p_{\text{intended}}(y|x)$ . This intended distribution captures the desired outputs across all possible inputs, reflecting the underlying goals.

However, $p_{\text{intended}}(y|x)$ is rarely something we can write down explicitly. Human intentions are multifaceted, often implicit, context-dependent, and sometimes even contradictory. We typically aim for qualities often summarized by frameworks like Anthropic's "Helpful, Honest, and Harmless" (HHH):

Helpful: The model should assist the user in achieving their stated or implied goals effectively.
Honest: The model should provide accurate information and avoid deception or fabrication. It should also express uncertainty appropriately.
Harmless: The model should not generate outputs that are harmful, unethical, toxic, discriminatory, or promote illegal activities.

While conceptually useful, translating these high-level principles into a concrete, optimizable objective function that works for a complex system like an LLM is a primary difficulty. We usually resort to proxy objectives:

Supervised Fine-Tuning (SFT): Maximize the likelihood of desired responses $(x, y_{\text{desired}})$ in a curated dataset. The proxy is the negative log-likelihood loss on this dataset. $\mathcal{L}_{\text{SFT}} = - \sum_{(x, y_{\text{desired}})} \log p_{\theta}(y_{\text{desired}}|x)$
Reinforcement Learning (RL): Maximize the expected reward $R(x, y)$ assigned by a reward model (RM), which itself is trained to predict human preferences. The proxy is the learned reward function $R_{\theta_{\text{RM}}}(x, y)$ . $\text{Maximize } \mathbb{E}_{x \sim \mathcal{D}, y \sim p_{\theta}(y|x)} [R_{\theta_{\text{RM}}}(x, y)]$

The gap between the true intended objective $p_{\text{intended}}$ and the proxy objective we can actually optimize is a fundamental source of alignment failures.

Core Challenges in Solving the Alignment Problem

Bridging the gap between intent and outcome involves navigating several significant technical hurdles:

Objective Specification: How do we translate vague human values and preferences into a precise specification that can guide model training?
- Ambiguity: Natural language instructions or preferences are often underspecified or open to multiple interpretations.
- Scalability of Supervision: Creating high-quality SFT data or preference labels requires significant human effort, making it difficult to cover the vast space of possible interactions.
- Implicit Goals: Users often have unstated assumptions or goals that the model should ideally infer and respect.
Optimization and Learning Dynamics: Even with a reasonable proxy objective, the optimization process itself can lead to unintended behaviors.
- Proxy Gaming: The model may find ways to maximize the proxy objective ( $R_{\text{proxy}}$ or minimizing $\mathcal{L}_{\text{proxy}}$ ) without actually satisfying the intended goal ( $R_{\text{intended}}$ or $p_{\text{intended}}$ ). This is closely related to specification gaming and reward hacking, which we'll detail later. For example, a model might learn to sound very confident to maximize a reward signal associated with perceived helpfulness, even when it's uncertain or incorrect.
- Inner Alignment Failure: The model might develop internal representations or "goals" during training that are misaligned with the specified outer objective. It might appear aligned during training or on standard evaluations but behave unpredictably or dangerously when faced with novel situations or distributional shifts. This means the model's internal reasoning diverges from the intended reasoning, even if its output seems correct on the surface for familiar inputs.
- Optimization Stability: Techniques like RLHF can be complex and unstable, requiring careful hyperparameter tuning and monitoring to avoid model collapse or undesirable policy updates.
Robustness and Generalization: Alignment achieved in a controlled training environment may not hold up in the real world.
- Distributional Shift: Models may encounter inputs or scenarios significantly different from their training data, potentially leading to degraded performance or safety failures.
- Adversarial Attacks: Malicious actors may craft inputs (jailbreaks, prompt injections) specifically designed to bypass safety mechanisms and elicit harmful or unintended outputs (covered in Chapter 5).
Evaluation: How can we reliably measure alignment and anticipate potential failures before deployment?
- Scalability of Evaluation: Exhaustively testing an LLM across all possible inputs and scenarios is intractable.
- Limitations of Benchmarks: Automated benchmarks (discussed in Chapter 4) capture specific aspects of alignment but may miss subtle failures or not reflect real-world usage patterns accurately. Human evaluation and red teaming are essential but costly and slow.
- "Unknown Unknowns": We might not even know what failure modes to look for, especially with highly capable future models.

The alignment problem visualized: bridging the gap between the ideal intended behavior and the actual behavior resulting from optimizing imperfect proxy objectives. Key challenges arise during specification (translating intent to a proxy) and optimization/generalization (ensuring the learned policy matches intent robustly).

Effectively addressing the alignment problem requires progress on all these fronts: developing better ways to specify intent, creating more robust optimization techniques that are less susceptible to proxy gaming, designing comprehensive and scalable evaluation methods, and building systems that are resilient to adversarial pressures and distributional shifts. The following chapters explore specific techniques aimed at tackling these challenges, starting with a deep dive into RLHF.