To train a reward model ( $RM$ ) effectively, we need a learning objective that aligns the model's outputs with the collected human preference data. Since our data consists of pairwise comparisons for a given prompt $x$ , indicating a preference for a "winning" response $y_w$ over a "losing" response $y_l$ , we aim to train the $RM$ to assign a higher scalar score to $y_w$ than to $y_l$ .

The standard approach draws inspiration from probabilistic choice models like the Bradley-Terry model. We model the probability that humans prefer $y_w$ over $y_l$ given the prompt $x$ as a function of the difference between the reward model's scores for each response. Specifically, we use the logistic function (sigmoid, $\sigma$ ) to map the score difference to a probability:

P(y_w \succ y_l | x) = \sigma(RM_\theta(x, y_w) - RM_\theta(x, y_l))

Here:

$RM_\theta(x, y)$ is the scalar score assigned by the reward model (with parameters $\theta$ ) to response $y$ given prompt $x$ .
$y_w \succ y_l$ denotes that response $y_w$ is preferred over response $y_l$ .
$\sigma(z) = \frac{1}{1 + e^{-z}}$ is the sigmoid function, which squashes its input to the range (0, 1), suitable for representing probabilities.

The goal of training is to find the parameters $\theta$ that maximize the likelihood of observing the preferences expressed in our dataset $D = \{(x^{(i)}, y_w^{(i)}, y_l^{(i)})\}$ . Maximizing the likelihood is equivalent to minimizing the negative log-likelihood of the preferences. For a single preference pair $(x, y_w, y_l)$ , the negative log-likelihood loss is:

\mathcal{L}(\theta; x, y_w, y_l) = -\log P(y_w \succ y_l | x)

Substituting the probability expression, we get:

\mathcal{L}(\theta; x, y_w, y_l) = -\log \sigma(RM_\theta(x, y_w) - RM_\theta(x, y_l))

This is often referred to as the pairwise logistic loss. To train the model, we minimize the average loss over the entire preference dataset $D$ :

\mathcal{L}_{total}(\theta) = -\frac{1}{|D|} \sum_{(x, y_w, y_l) \in D} \log \sigma(RM_\theta(x, y_w) - RM_\theta(x, y_l))

Minimizing this loss function using gradient-based optimization methods (like Adam) encourages the reward model to assign a higher score to the preferred response $y_w$ compared to the less preferred response $y_l$ . The larger the difference $RM_\theta(x, y_w) - RM_\theta(x, y_l)$ , the lower the loss for that specific pair, pushing the model towards correctly ranking the responses according to human judgments.

The following diagram illustrates the computation flow for a single preference pair during training:

Computation flow for calculating the pairwise logistic loss for a single preference sample $(x, y_w, y_l)$ . The reward model $RM_\theta$ computes scalar scores for both responses given the prompt. The difference in scores is passed through a sigmoid function, and the negative logarithm of this probability forms the loss contribution for this sample. This loss is then used to update the model parameters $\theta$ via backpropagation.

This training objective directly translates the pairwise human preferences into a gradient signal that shapes the reward model. A well-trained RM, optimized using this objective, provides the crucial reward signal needed for the subsequent reinforcement learning phase, guiding the language model towards generating responses that align better with human preferences.