Preference Dataset Formats and Structures

To effectively train a reward model, a consistent way to structure human feedback data is necessary. The goal is to represent the outcome of pairwise comparisons, indicating which response was preferred for a given input prompt.

The fundamental unit of data for reward modeling typically involves a triplet: the input prompt ($x$), the response deemed better or "chosen" ($y_w$), and the response deemed worse or "rejected" ($y_l$). This structure directly feeds into the loss function based on the Bradley-Terry model or similar comparison frameworks.

Common Preference Dataset Structures

1. Pairwise Comparison Tuples: This is the most prevalent format. Each data point explicitly represents a single comparison judgment.

   • Structure: (prompt, chosen_response, rejected_response)
   • Example Record: ("Explain RLHF.", "RLHF uses human feedback to train a reward model...", "RLHF is about reinforcement learning.")
   • Usage: This format directly maps to the reward model's training objective, where the model learns to assign a higher score to chosen_response than rejected_response given the prompt.

2. Ranked Lists (Convertible to Pairwise): Sometimes, annotators might rank multiple responses generated for the same prompt (e.g., Best > Good > Fair > Bad).

   • Structure: (prompt, [ranked_response_1, ranked_response_2, ..., ranked_response_n]) where the order indicates preference (1 is best).
   • Conversion: A ranked list of $n$ responses can be decomposed into $\binom{n}{2}$ pairwise comparisons. For example, a ranking [A, B, C] (where A is best) implies the pairs (A, B), (A, C), and (B, C).
   • Considerations: While richer, ranking multiple items can be more cognitively demanding for annotators than simple pairwise choices. The decomposition assumes transitivity of preferences.

3. Grouped Preferences: Datasets might group all comparisons related to a single prompt.

   • Structure: A dictionary or object where the key is the prompt and the value is a list of (chosen_response, rejected_response) pairs.
   • Benefit: Can be convenient for analysis and ensuring all comparisons for a prompt are processed together.

Example: Anthropic HH-RLHF Dataset Structure

A widely referenced dataset is Anthropic's "Helpful and Harmless Reinforcement Learning from Human Feedback" (HH-RLHF). It primarily uses the pairwise comparison format. Each entry contains:

• A prompt (often the start of a conversation).
• A chosen completion (the response preferred by the human labeler).
• A rejected completion (the response not preferred).

This clean structure makes it straightforward to apply the standard reward modeling loss function.

Data Representation Example

Here’s a simplified diagram illustrating the core relationship in a pairwise preference record:

A single preference data point links a prompt to two responses, explicitly identifying the preferred (chosen) and non-preferred (rejected) options.

Tabular Representation

For processing, these preferences are often organized into tables or structured files (like JSON Lines or CSV).

Prompt	Chosen Response	Rejected Response
"Summarize the process of photosynthesis."	"Plants use sunlight, water, and CO2 to create..."	"Photosynthesis is how plants make food."
"Write a short poem about a cat."	"Soft paws tread lightly,\nWhiskers twitch..."	"Cats are furry,\nThey like to sleep a lot."
"Explain the concept of KL divergence."	"KL divergence measures the difference between..."	"It's a math thing for distributions."

Metadata and Insights

The core triplet, practical datasets often include metadata:

• Annotator ID: To track labeler consistency or potential biases.
• Timestamp: When the annotation was made.
• Annotation Task ID: Linking to specific instructions or batches.
• Reasoning (Optional): Sometimes annotators provide free-text reasons for their choice, which can be valuable for analysis but isn't typically used directly in standard RM training.
• Margin (Optional): Some collection interfaces allow annotators to specify the degree of preference (e.g., slightly better, much better). This can inform weighted loss functions but adds complexity.

Preparing Data for the Reward Model

When training the reward model, the prompt and each response (chosen and rejected) are typically concatenated and tokenized. For instance, the input to the reward model to score the chosen response might look like [tokenizer.bos_token] + tokenize(prompt) + tokenize(chosen_response) + [tokenizer.eos_token]. The model is trained to output a scalar score for such combined sequences. The difference between the scores for the (prompt, chosen_response) pair and the (prompt, rejected_response) pair is then used in the loss calculation, as shown in the chapter introduction's formula:

$\text{loss} = -\log(\sigma(RM(x, y_w) - RM(x, y_l)))$

Understanding these data formats is essential for correctly preparing preference datasets and implementing the reward modeling stage of the RLHF pipeline. The choice of format impacts data collection interfaces, storage, and the preprocessing steps required before feeding the data into the reward model training loop.