Calibration of Reward Models

Reward models (RM) are trained with an objective, typically based on the Bradley-Terry framework, to effectively learn relative preferences between pairs of responses. However, the raw output scores $RM(\text{prompt}, \text{response})$ from such models do not automatically possess a meaningful scale. The difference between two scores, $RM(\text{prompt}, \text{response}_1) - RM(\text{prompt}, \text{response}_2)$, determines the predicted probability $P(\text{response}_1 \succ \text{response}_2)$ via the sigmoid function. Yet, does a score difference of 2 represent twice as "strong" a preference as a difference of 1? Not necessarily.

Calibration addresses this issue. A well-calibrated reward model produces scores where the predicted probabilities accurately reflect the true probabilities observed in the human preference data. If the model predicts $\sigma(RM_1 - RM_2) = 0.8$, we expect that for pairs with approximately this score difference, humans indeed preferred response 1 around 80% of the time. Without calibration, the RM might be systematically overconfident (e.g., predicting 0.9 probability when the actual preference rate is only 70%) or underconfident.

Assessing Reward Model Calibration

A standard technique for evaluating calibration is using a reliability diagram (also known as a calibration plot).

1. Binning: Divide the range of predicted preference probabilities (e.g., [0, 1]) or raw score differences into several bins (e.g., 10 bins).
2. Calculating Averages: For all preference pairs $(y_w, y_l)$ falling into a specific bin based on their predicted probability $\sigma(RM(y_w) - RM(y_l))$:
   • Calculate the average predicted probability within that bin.
   • Calculate the actual fraction of pairs where $y_w$ was indeed preferred (this should always be 1 by definition if using $(y_w, y_l)$ pairs, but it's useful if assessing on raw score differences or a separate test set where the outcome isn't fixed by the pair order). A better approach is to compute the average accuracy within the bin: the proportion of pairs correctly classified according to the preference label. For a bin centered around probability $p$, the expected accuracy is $p$.
3. Plotting: Plot the average observed accuracy (or fraction of positives) against the average predicted probability for each bin.

Perfect calibration corresponds to the $y=x$ line. Deviations indicate miscalibration: points below the line indicate overconfidence (predicted probability > actual accuracy), while points above indicate underconfidence.

Example reliability diagram comparing an uncalibrated model (overconfident at higher probabilities) and a better-calibrated model.

Quantitatively, calibration can be measured using metrics like the Expected Calibration Error (ECE), which computes the weighted average difference between predicted probabilities and observed accuracies across bins.

Techniques for Improving Calibration

If assessment reveals poor calibration, several techniques can be applied:

1. Temperature Scaling: This is a simple and often effective post-processing method. It involves rescaling the logits (the inputs to the final activation function, in our case, the raw RM scores or their difference) by a learned temperature parameter $T > 0$. The calibrated probability is calculated as: $P_{calibrated}(\text{response}_1 \succ \text{response}_2) = \sigma\left( \frac{RM(\text{prompt}, \text{response}_1) - RM(\text{prompt}, \text{response}_2)}{T} \right)$ The temperature $T$ is optimized on a hold-out validation set of preference pairs. The goal is typically to minimize the negative log-likelihood (NLL) or a calibration metric like ECE on this validation set.

   • If $T > 1$, it "cools" the logits, softening probabilities and reducing model confidence (correcting overconfidence).
   • If $T < 1$, it "heats" the logits, sharpening probabilities and increasing confidence (correcting underconfidence).
   • $T=1$ leaves the probabilities unchanged. Temperature scaling only adjusts the confidence of the predictions without changing the ranking (argmax) of the outputs, making it a safe adjustment.

2. Label Smoothing: Applied during the initial RM training phase, label smoothing replaces hard targets (0 or 1) with slightly softened targets (e.g., 0.05 and 0.95). This discourages the model from producing extremely high-confidence predictions (pushing logits towards positive or negative infinity) and can implicitly improve calibration.

3. Isotonic Regression: Another post-processing technique that fits a non-decreasing function to map the model's output probabilities to calibrated probabilities. It's more powerful than temperature scaling but requires more data and can sometimes be less stable.

4. Data Quality and Diversity: Fundamentally, calibration issues can arise from noisy labels, insufficient data in certain regions of the score space, or a mismatch between the training data distribution and the data encountered later. Improving the quality and diversity of the human preference dataset is always beneficial.

Why Calibration Matters for RL Fine-Tuning

The subsequent RL phase (typically using PPO) relies heavily on the reward signal provided by the RM. The magnitude of the reward $r = RM(\text{prompt}, \text{response})$ or the advantage calculated using it directly influences the scale of policy updates.

• Overconfident RM: An overconfident RM might assign excessively high rewards for responses it slightly prefers, leading the RL policy to optimize too aggressively towards specific response types that might only be marginally better, potentially sacrificing diversity or exploring less. It can also cause instability if small changes in generated text lead to large swings in reward.
• Underconfident RM: An underconfident RM might provide weak reward signals, making it difficult for the RL agent to distinguish meaningfully between good and slightly less good responses, slowing down learning or preventing the policy from reaching optimal alignment.

A well-calibrated RM provides a more reliable and interpretable reward signal. The magnitude of the reward difference between two potential responses better reflects the actual strength of human preference, leading to more stable and effective policy updates during the RL fine-tuning stage. While PPO often involves normalizing advantages, the relative scale of rewards derived from a calibrated model is still beneficial for learning behaviors aligned with human judgments. Therefore, assessing and improving RM calibration is an important step before proceeding to the RL optimization phase.