Evaluating SFT Models on Alignment Goals

Evaluating Supervised Fine-Tuning (SFT) models focuses on verifying if a model has become better aligned with desired behaviors, distinguishing its purpose from the goals of initial pre-training. This evaluation assesses the model's capability to follow instructions, provide helpful responses, and adhere to specified constraints, rather than raw language modeling ability (such as perplexity on a general corpus). This assessment step is important because it confirms the success of the SFT phase and often informs subsequent alignment stages like Reinforcement Learning from Human Feedback (RLHF).

Defining Alignment Goals for Evaluation

Before evaluating, clearly define the specific alignment goals SFT aimed to achieve. These typically include:

1. Instruction Following: Can the model accurately interpret and execute diverse instructions provided in the prompt? This is the core objective for many SFT datasets.
2. Helpfulness: Does the model generate responses that are useful, relevant, and informative in the context of the prompt?
3. Format Adherence: If the instructions specify a particular output format (e.g., JSON, markdown list, specific tone), does the model comply?
4. Safety and Harmlessness: While often a major focus of RLHF, SFT evaluation should also check if the model avoids generating harmful, biased, or inappropriate content, and whether SFT introduced any regressions in this area.

Measuring these qualities often requires moving past standard automated metrics.

Human Evaluation: The Ground Truth

For subjective attributes like helpfulness, instruction following fidelity on complex tasks, and harmlessness, human evaluation remains the most reliable method. Setting up effective human evaluation involves several considerations:

• Task Design: Create evaluation prompts that specifically probe the targeted alignment goals. These prompts might be drawn from a held-out set of the SFT data, newly created prompts, or established alignment benchmarks.
• Evaluation Rubrics: Develop clear and consistent scoring guidelines (rubrics). This could involve Likert scales (e.g., 1-5 rating for helpfulness), pairwise comparisons (Which response is better, A or B?), or categorical judgments (Did the model follow the format instruction: Yes/No?).
• Rater Training: Ensure evaluators understand the task, the rubric, and potential biases. Consistency checks among raters are important.
• Sampling: Evaluating every possible input is impossible. Select a representative sample of prompts covering various instruction types, complexities, and potential failure modes.

While powerful, human evaluation is resource-intensive (time, cost) and can suffer from inter-annotator disagreement. It's often used to validate automated metrics or for periodic deep assessments.

Workflow for evaluating SFT model responses using both human and automated methods.

Automated Evaluation Methods

To complement human evaluation and enable faster iteration, several automated approaches are used:

1. Model-Based Evaluation: Leverage a powerful, pre-existing LLM (often referred to as an "evaluator model," e.g., GPT-4, Claude) to assess the quality of the SFT model's responses.

   • Process: Craft a prompt for the evaluator model, providing it with the original instruction, the SFT model's response, and potentially a reference answer or a clear rubric. The evaluator model then outputs a score or a judgment.
   • Example Prompt for Evaluator LLM:

     You are an impartial judge evaluating the quality of an AI assistant's response to a user's instruction.
     Instruction: "Summarize the following text into three bullet points:\n[Long text snippet here...]"
     Assistant's Response: "[Model's generated summary here...]"
     
     Evaluate the response based on the following criteria:
     1. Accuracy: Does the summary accurately reflect the main points of the original text? (1-5)
     2. Conciseness: Is the summary brief and to the point? (1-5)
     3. Format Compliance: Did the assistant use exactly three bullet points? (Yes/No)
     
     Provide your ratings in JSON format: {"accuracy": <score>, "conciseness": <score>, "format_compliance": "<yes/no>"}
     Also provide a brief justification for your scores.
     

   • Pros: Scalable, faster than human evaluation. Can capture nuances missed by simpler metrics.
   • Cons: Dependent on the quality and potential biases of the evaluator model. Susceptible to prompt sensitivity. Can be computationally expensive. Calibration against human judgments is often necessary.

2. Benchmark Datasets: Evaluate the SFT model on established benchmarks designed specifically for instruction following or helpfulness. Examples include:

   • AlpacaEval: Uses GPT-4 to automatically compare model outputs against reference responses (e.g., from text-davinci-003) on the Alpaca instruction set.
   • MT-Bench: A multi-turn benchmark assessing conversational and instruction-following abilities across different categories, often judged by strong LLMs.
   • HELM (Holistic Evaluation of Language Models): While broader, HELM includes specific scenarios and metrics relevant to alignment goals like instruction following and robustness.

   Using benchmarks involves running the SFT model on the benchmark prompts and then using the benchmark's prescribed evaluation protocol (often model-based or human-based).

3. Reference-Based Metrics (Use with Caution): Metrics like ROUGE (for summarization) or BLEU (for translation) can be used if the SFT task involves generating text that should closely match a reference (e.g., fine-tuning for a specific summarization style). However, they are often poor indicators of general instruction following or helpfulness because:

   • They penalize valid paraphrasing and stylistic variations.
   • They don't measure factual correctness or logical reasoning.
   • A high overlap score doesn't guarantee the core instruction was followed.

   Consider this simple example: Instruction: "Explain gravity in one sentence." Reference: "Gravity is the force by which a planet or other body draws objects toward its center." Model A: "Gravity is the fundamental force attracting objects with mass towards each other." (Good, low BLEU/ROUGE) Model B: "Gravity is a force. Objects get drawn by planets to their centers." (Okay, higher BLEU/ROUGE but less fluent)

   Using ROUGE here might misleadingly favor Model B.

Implementing Evaluation

Let's illustrate fetching a response and preparing for evaluation. Assume you have loaded your SFT model and tokenizer using PyTorch and Hugging Face transformers.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Assume model and tokenizer are loaded
# model = AutoModelForCausalLM.from_pretrained(
#     "path/to/your/sft_model"
# )
# tokenizer = AutoTokenizer.from_pretrained(
#     "path/to/your/sft_model"
# )
# device = torch.device(
#     "cuda" if torch.cuda.is_available() else "cpu"
# )
# model.to(device)
# model.eval() # Set model to evaluation mode

# --- Placeholder for actual model loading ---
class DummyModel: # Simulate a loaded model
    def generate(
        self, input_ids, attention_mask, max_new_tokens, pad_token_id
    ):
        # Simulate generation based on input length
        new_tokens = torch.randint(
            100,
            1000,
            (input_ids.shape[0], max_new_tokens),
            device=input_ids.device
        )
        output_ids = torch.cat([input_ids, new_tokens], dim=1)
        return output_ids
class DummyTokenizer: # Simulate a loaded tokenizer
    def __init__(self):
        self.pad_token_id = 0
    def encode(self, text, return_tensors=None):
        # Very simple simulation
        tokens = [101] + [ i+1000 for i in range(len(text.split())) ]
        return torch.tensor([tokens], dtype=torch.long)
    def decode(self, ids, skip_special_tokens=False):
        # Very simple simulation
        words = [
            f"word{i-1000}" if i >= 1000 else "[CLS]"
            for i in ids[0].tolist()
        ]
        return " ".join(words)
    def __call__(
        self, text, return_tensors=None, padding=False, truncation=False
    ):
         # Simulate __call__ used often
        encoded = self.encode(text, return_tensors)
        return {"input_ids": encoded, "attention_mask": torch.ones_like(encoded)}

model = DummyModel()
tokenizer = DummyTokenizer()
device = torch.device("cpu") # Simplified for example
# --- End Placeholder ---

def generate_response(prompt_text, model, tokenizer, max_new_tokens=100):
    """Generates a response from the SFT model."""
    inputs = tokenizer(
        prompt_text, return_tensors="pt", padding=True, truncation=True
    ).to(device)

    with torch.no_grad():
        output_ids = model.generate(
            inputs.input_ids,
            attention_mask=inputs.attention_mask,
            max_new_tokens=max_new_tokens,
            pad_token_id=tokenizer.pad_token_id
        )

    # Decode only the newly generated tokens
    input_length = inputs.input_ids.shape[1]
    generated_ids = output_ids[:, input_length:]
    response = tokenizer.decode(generated_ids, skip_special_tokens=True)
    return response

# Example evaluation prompt
eval_prompt = (
    "Instruction: Write a short python function to calculate factorial."
    "\nResponse:"
)
# Note: Good SFT formatting includes clear separators like
# \nResponse:

generated_text = generate_response(eval_prompt, model, tokenizer)

print(f"Evaluation Prompt:\n{eval_prompt}")
print(f"\nGenerated Response:\n{generated_text}")

# --- Next Steps would be: ---
# 1. Send `eval_prompt` and `generated_text` to human evaluators.
# 2. Or, format them for a model-based evaluator (like GPT-4).
# 3. Or, if part of a benchmark, use the benchmark's specific evaluation
#    script.
# 4. Or, apply simpler checks (e.g., check if 'def' and 'return' are
#    in generated_text) - limited but fast.

Comparing Evaluation Methods

Different evaluation methods provide different signals. It's often useful to combine them. For instance, use automated metrics/benchmarks for broad coverage and frequent checks, and use human evaluation periodically to validate the automated results and probe for subtle issues.

Comparison of evaluation scores across different methods for assessing SFT model alignment. Note how automated checks might differ from human or model-based ratings.

Ultimately, evaluating SFT models effectively requires a clear understanding of the alignment goals and a thoughtful combination of human insight and scalable automated techniques. The results guide further fine-tuning efforts, helping to create LLMs that are not just capable, but genuinely helpful and reliable.