Training Objectives for Reward Models

To train a reward model ( $RM$ ) effectively, a learning objective is needed that aligns the model's outputs with the collected human preference data. This data commonly consists of pairwise comparisons for a given prompt $x$ , indicating a preference for a "winning" response $y_w$ over a "losing" response $y_l$ . The objective is to train the $RM$ to assign a higher scalar score to $y_w$ than to $y_l$ .

The standard approach draws inspiration from probabilistic choice models like the Bradley-Terry model. We model the probability that humans prefer $y_w$ over $y_l$ given the prompt $x$ as a function of the difference between the reward model's scores for each response. Specifically, we use the logistic function (sigmoid, $\sigma$ ) to map the score difference to a probability:

P(y_w \succ y_l | x) = \sigma(RM_\theta(x, y_w) - RM_\theta(x, y_l))

Here:

$RM_\theta(x, y)$ is the scalar score assigned by the reward model (with parameters $\theta$ ) to response $y$ given prompt $x$ .
$y_w \succ y_l$ denotes that response $y_w$ is preferred over response $y_l$ .
$\sigma(z) = \frac{1}{1 + e^{-z}}$ is the sigmoid function, which squashes its input to the range (0, 1), suitable for representing probabilities.

The goal of training is to find the parameters $\theta$ that maximize the likelihood of observing the preferences expressed in our dataset $D = \{(x^{(i)}, y_w^{(i)}, y_l^{(i)})\}$ . Maximizing the likelihood is equivalent to minimizing the negative log-likelihood of the preferences. For a single preference pair $(x, y_w, y_l)$ , the negative log-likelihood loss is:

\mathcal{L}(\theta; x, y_w, y_l) = -\log P(y_w \succ y_l | x)

Substituting the probability expression, we get:

\mathcal{L}(\theta; x, y_w, y_l) = -\log \sigma(RM_\theta(x, y_w) - RM_\theta(x, y_l))

This is often referred to as the pairwise logistic loss. To train the model, we minimize the average loss over the entire preference dataset $D$ :

\mathcal{L}_{total}(\theta) = -\frac{1}{|D|} \sum_{(x, y_w, y_l) \in D} \log \sigma(RM_\theta(x, y_w) - RM_\theta(x, y_l))

Minimizing this loss function using gradient-based optimization methods (like Adam) encourages the reward model to assign a higher score to the preferred response $y_w$ compared to the less preferred response $y_l$ . The larger the difference $RM_\theta(x, y_w) - RM_\theta(x, y_l)$ , the lower the loss for that specific pair, pushing the model towards correctly ranking the responses according to human judgments.

The following diagram illustrates the computation flow for a single preference pair during training:

Computation flow for calculating the pairwise logistic loss for a single preference sample $(x, y_w, y_l)$ . The reward model $RM_\theta$ computes scalar scores for both responses given the prompt. The difference in scores is passed through a sigmoid function, and the negative logarithm of this probability forms the loss contribution for this sample. This loss is then used to update the model parameters $\theta$ via backpropagation.

This training objective directly translates the pairwise human preferences into a gradient signal that shapes the reward model. A well-trained RM, optimized using this objective, provides the important reward signal needed for the subsequent reinforcement learning phase, guiding the language model towards generating responses that align better with human preferences.

Was this section helpful?

References

Learning to summarize with human feedback, Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano, 2020 Advances in Neural Information Processing Systems (NeurIPS) DOI: 10.48550/arXiv.2009.01325 - An earlier work from OpenAI demonstrating the use of human feedback for training reward models, specifically for summarization tasks, employing a similar pairwise preference learning objective.
Rank analysis of incomplete block designs. I. The method of paired comparisons, Ralph Allan Bradley and Milton E. Terry, 1952 Biometrika, Vol. 39 (Biometrika Trust) DOI: 10.1093/biomet/39.3-4.324 - The foundational paper introducing the Bradley-Terry model, which provides the probabilistic framework for pairwise comparisons and serves as the theoretical inspiration for the reward model's loss function.
Deep Reinforcement Learning from Human Preferences, Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei, 2017 Advances in Neural Information Processing Systems (NeurIPS) DOI: 10.48550/arXiv.1706.03741 - A seminal work that proposed using human feedback, specifically pairwise comparisons, to train reward models for general deep reinforcement learning agents, predating its widespread application to language models.