Curating High-Quality SFT Datasets

The Supervised Fine-Tuning (SFT) phase enables successful Reinforcement Learning from Human Feedback (RLHF). Its purpose is to adapt a general pre-trained language model to the specific style, format, and domain relevant to your alignment goals before introducing reinforcement learning. The quality of the dataset used for SFT directly influences the initial policy model's capabilities and, consequently, the effectiveness and efficiency of the entire RLHF pipeline. A well-curated SFT dataset provides a strong starting point, teaching the model the desired baseline behavior through demonstration.

Sources of SFT Data

Obtaining high-quality prompt-response pairs is central to SFT. There are several common approaches, each with its own trade-offs:

1. Human Demonstrations: This is often considered the gold standard. Human writers, following specific guidelines, craft both the prompts and the desired responses.

   • Pros: High potential for quality and direct alignment with instructions. Allows fine-grained control over the desired behavior.
   • Cons: Can be expensive and time-consuming to scale. Requires clear, detailed instructions and rigorous quality control to ensure consistency and minimize annotator bias. The quality is heavily dependent on the skill and training of the human annotators.

2. Filtered Existing Datasets: Leveraging publicly available instruction-following or dialogue datasets (e.g., instruction datasets derived from StackExchange, curated academic sets, or datasets like OpenAssistant).

   • Pros: Can provide a large volume of data relatively quickly and at lower cost. May offer broad topic coverage.
   • Cons: Quality can be highly variable. Data may not perfectly match the target domain, style, or safety requirements. Requires significant effort in filtering, cleaning, and potentially reformatting to select only high-quality, relevant examples. May contain biases or undesired behaviors present in the original source.

3. Model-Generated Data with Human Review/Editing: Using a capable existing LLM (sometimes called a "teacher" model) to generate initial responses to prompts, which are then reviewed, rated, and edited by humans.

   • Pros: Can accelerate data generation compared to pure human writing. Humans focus on refinement rather than initial drafting.
   • Cons: Risks inheriting biases or stylistic quirks of the generator model. Still requires significant human effort for review and quality assurance. The generator model might not be capable of producing the desired level of quality or adherence to specific constraints without careful prompting and filtering.

Regardless of the source, the emphasis must be on quality and relevance to the target alignment objectives. A smaller, high-quality dataset is often more effective for SFT than a massive, noisy one.

Characteristics of High-Quality SFT Data

What constitutes a "high-quality" demonstration? Several attributes are significant:

• Alignment: Examples must directly reflect the desired traits (e.g., helpfulness, harmlessness, honesty, specific tone, adherence to instructions). If the goal is harmlessness, the SFT data should not contain harmful examples, even if the prompts might invite them.
• Instruction Following: Responses should accurately and completely address the prompt. For complex prompts, the response should demonstrate an understanding of the different parts of the request.
• Clarity and Coherence: Responses should be well-written, grammatically correct, easy to understand, and logically structured.
• Factual Accuracy (where applicable): For prompts requiring factual information, the responses should be accurate and avoid making things up (hallucination).
• Appropriate Format and Style: If the model is expected to produce outputs in specific formats (like markdown, JSON, or code) or adopt a particular persona or style, the SFT examples must consistently demonstrate this.
• Diversity: The dataset should cover a wide spectrum of anticipated user inputs, topics, and instruction types within the target domain. This includes:
  • Varying prompt lengths and complexity.
  • Different user intentions (e.g., asking questions, giving commands, seeking creative content).
  • Potentially challenging or ambiguous prompts.
  • Examples covering relevant edge cases.

A typical workflow for curating SFT datasets involves defining requirements, sourcing data, filtering or creating examples, performing quality checks, cleaning the data, and potentially refining the guidelines based on review feedback.

Curation Process and Best Practices

Creating an effective SFT dataset requires a systematic approach:

1. Develop Clear Guidelines: Create a detailed specification document for annotators or for filtering criteria. This should explicitly define the desired response characteristics, style, tone, formatting rules, and what constitutes a "good" versus "bad" example. Include examples and counterexamples.
2. Iterative Refinement: Start with a smaller batch of data. Train an initial SFT model and evaluate its performance qualitatively and quantitatively. Use the findings to refine the guidelines and curation process before scaling up data collection or filtering. Quality often matters more than sheer quantity, especially early on.
3. Implement Rigorous Quality Control (QC):
   • Use multiple annotators for some portion of the data to measure inter-annotator agreement.
   • Have experienced reviewers or domain experts check samples.
   • Track annotator performance and provide feedback.
   • Automate checks where possible (e.g., code linting, format validation).
4. Address Potential Biases: Be mindful of demographic, cultural, political, or stylistic biases potentially introduced by annotators or source data. Actively seek diversity in prompts and expected responses, and explicitly instruct annotators to avoid harmful stereotypes or biased language.
5. Standardize Data Format: Ensure all examples follow a consistent structure, often a JSON format containing fields for prompt, response, and potentially metadata like source, annotator_id, or quality_rating.

[
  {
    "prompt": "Explain the concept of KL divergence in the context of RLHF PPO in simple terms.",
    "response": "In RLHF PPO, KL divergence measures how much the model's behavior (policy) changes during training compared to its original behavior (before PPO tuning, often the SFT model). We add a penalty based on this divergence to the PPO objective. \n\nThink of it like a leash: we want the model to learn from the reward signal (get better according to human preferences), but we don't want it to stray *too far* or *too quickly* from the generally sensible behavior it learned during SFT. Too much divergence can lead to unpredictable or nonsensical outputs, or finding loopholes ('reward hacking'). The KL penalty helps keep the training stable and ensures the model doesn't completely forget its initial training.",
    "source": "human_annotator_v2",
    "category": "RL_concepts"
  },
  {
    "prompt": "Write a Python function to calculate the factorial of a non-negative integer.",
    "response": "```python\ndef factorial(n):\n  \"\"\"Calculates the factorial of a non-negative integer.\n\n  Args:\n    n: A non-negative integer.\n\n  Returns:\n    The factorial of n.\n\n  Raises:\n    ValueError: If n is negative.\n    TypeError: If n is not an integer.\n  \"\"\"\n  if not isinstance(n, int):\n    raise TypeError(\"Input must be an integer.\")\n  if n < 0:\n    raise ValueError(\"Input must be non-negative.\")\n  if n == 0:\n    return 1\n  else:\n    res = 1\n    for i in range(1, n + 1):\n      res *= i\n    return res\n\n# Example usage:\nprint(factorial(5)) # Output: 120\n```",
    "source": "filtered_code_dataset_v1",
    "category": "coding"
  }
]

Example structure for SFT data points, often stored in JSON or similar formats. Consistency is important for parsing during training.

Challenges in SFT Dataset Curation

Curating these datasets is not without difficulties:

• Cost and Scalability: High-quality human annotation is expensive. Scaling to hundreds of thousands or millions of examples requires significant investment and infrastructure.
• Subjectivity and Disagreement: For stylistic elements or instructions, different annotators might have varying interpretations, leading to inconsistency. Clear guidelines and calibration sessions are needed.
• Implicit Bias: Even with explicit instructions against bias, subtle biases can creep into the data based on the annotator pool or the source material. Ongoing audits and diverse perspectives in the curation team can help mitigate this.
• Ensuring Diversity: It's challenging to anticipate the full range of user interactions. Datasets might inadvertently lack coverage for specific topics, user types, or edge cases, leading to unexpected model failures later.
• Maintaining Quality at Scale: As dataset size increases, ensuring consistent quality across all examples becomes harder. Quality control processes are essential.

The effort invested in curating a high-quality SFT dataset pays dividends throughout the RLHF process. A well-initialized model provides a stronger foundation for reward modeling, requires less drastic policy updates during RL fine-tuning (often leading to more stable PPO training), and ultimately contributes to a better-aligned final language model. Neglecting SFT data quality can lead to difficulties in subsequent stages, potentially requiring more extensive preference data or more complex RL tuning strategies to achieve the desired alignment goals.