Concept of Learning from Preferences

Supervised Fine-Tuning aligns a model to mimic demonstrations, but it doesn't explicitly teach the model what makes one response better than another according to human values. To instill this sense of quality, we need a different approach. Instead of asking humans for absolute scores for generated text, which can be difficult, inconsistent, and subjective, RLHF relies on a more intuitive form of feedback: pairwise comparisons.

Humans are generally better at stating which of two options they prefer rather than assigning a precise numerical score to each. Think about judging essays, art, or even simple choices. It's often easier to say "A is better than B" than it is to assign A a score of 8.5 and B a score of 6.2 with confidence and consistency. Learning from preferences uses this human capability.

The Underlying Assumption: Preferences Imply Scores

The core idea is that human preferences between pairs of responses, given the same prompt, implicitly reveal an underlying reward function. If a human consistently prefers response $y_1$ over response $y_2$ for a given prompt $x$, it suggests that $y_1$ possesses more of the desired qualities (helpfulness, harmlessness, accuracy, etc.) than $y_2$. We assume there's a latent scalar function, the Reward Model ($RM$), denoted $RM(x, y)$, that assigns a score reflecting these desired qualities. The preference $y_1 \succ y_2$ (read as "$y_1$ is preferred over $y_2$") suggests that $RM(x, y_1) > RM(x, y_2)$.

Our goal is to train a model, typically a neural network based on the same architecture as the language model being tuned, to approximate this latent reward function. This $RM$ takes the prompt and a response as input and outputs a scalar value representing the predicted human preference score.

From Preferences to Probabilities

How do we train such a model using only comparison data? We frame the learning problem probabilistically. Models like the Bradley-Terry model (or variations) provide a mathematical link between pairwise comparisons and the underlying scores. As introduced in the chapter overview, we model the probability that a human prefers response $y_1$ over $y_2$ given prompt $x$ as a function of the difference between their respective reward model scores:

$$P(y_1 \succ y_2 | x) = \sigma(RM(x, y_1) - RM(x, y_2))$$

Here, $\sigma$ is the sigmoid function, $\sigma(z) = 1 / (1 + e^{-z})$.

Let's unpack this formula:

1. Score Difference: $RM(x, y_1) - RM(x, y_2)$ calculates the difference between the predicted scores for the two responses. A larger positive difference means the model predicts a stronger preference for $y_1$.
2. Sigmoid Function: The sigmoid function maps this difference, which can range from $-\infty$ to $+\infty$, into a probability between 0 and 1.
   • If $RM(x, y_1)$ is significantly larger than $RM(x, y_2)$, the difference is large and positive, so $\sigma(\text{difference})$ approaches 1. This means the model predicts a high probability that $y_1$ is preferred.
   • If $RM(x, y_2)$ is significantly larger than $RM(x, y_1)$, the difference is large and negative, so $\sigma(\text{difference})$ approaches 0. This means the model predicts a low probability that $y_1$ is preferred (or equivalently, a high probability that $y_2$ is preferred).
   • If the scores are equal, the difference is 0, and $\sigma(0) = 0.5$, indicating indifference or equal preference likelihood.

This probabilistic framing allows us to use standard machine learning techniques, specifically binary cross-entropy loss, to train the $RM$. The training data consists of tuples $(x, y_1, y_2)$, where $y_1$ is the preferred ("winning") response and $y_2$ is the non-preferred ("losing") response according to human labelers. The model learns to adjust its parameters such that it assigns higher scores to winning responses and lower scores to losing responses for a given prompt.

Diagram illustrating how a prompt and two responses lead to a human preference label, which serves as the target for training the Reward Model. The RM processes both responses to predict scores, their difference, and ultimately the probability of one being preferred over the other.

Advantages of Preference-Based Learning

Learning from preferences offers several advantages over directly specifying or regressing onto absolute reward scores:

1. Human Reliability: As mentioned, comparative judgments are often more reliable and consistent for humans than assigning absolute scores, especially for complex, multi-faceted criteria like the quality of generated text.
2. Data Efficiency: While collecting preferences still requires significant human effort, it can sometimes be more efficient than obtaining detailed feedback or perfect demonstration data needed for alternative approaches.
3. Implicit Normalization: The comparative nature automatically handles variations in individual raters' scoring scales. Whether one rater scores responses between 6-8 and another between 3-5 doesn't matter as much as whether they consistently agree on which response is better. The model learns the relative ordering.

By training a model to predict these pairwise preferences, we create a reward signal that reflects complex human judgments. This learned $RM$ becomes the objective function that the language model will later be optimized against using reinforcement learning techniques like PPO, guiding it towards generating responses that align better with human expectations. The next sections explore the practicalities of gathering this preference data and training the reward model itself.