After fine-tuning the base language model with your curated dataset of prompt-response pairs, the next step isn't immediately jumping into reward modeling. It's essential to first assess the quality and behavior of the Supervised Fine-Tuned (SFT) model. This evaluation serves as a critical checkpoint: did the SFT process successfully adapt the model towards the desired style, format, and instruction-following capabilities demonstrated in the training data? A poorly performing SFT model will provide a weak foundation for the subsequent reinforcement learning phase, potentially hindering the entire alignment process.

Evaluating the SFT model focuses on determining if it has internalized the characteristics of the demonstration data. We are less concerned with broad, general knowledge (which comes from pre-training) and more interested in its ability to generate responses that look like the high-quality examples it was trained on. Key questions include:

Does the model follow instructions accurately for prompts similar to those in the SFT dataset?
Does the model adopt the desired tone, style, and output format?
Has the fine-tuning process introduced undesirable behaviors like increased repetition, blandness, or refusal to answer reasonable prompts?
Is the model performing significantly better on target tasks compared to the base pre-trained model?

Answering these questions requires a combination of quantitative metrics and qualitative human assessment.

Evaluation Datasets

Just as in standard machine learning practice, evaluation should be performed on a held-out dataset (a validation or test set) that was not used during SFT training. This dataset should ideally contain prompts representative of the tasks and style you fine-tuned for. Using prompts from the training set would only measure memorization, not generalization.

Automated Metrics

While automated metrics often fall short of capturing the full picture of language quality and alignment, they can provide useful signals, especially for identifying training issues.

Perplexity

Perplexity is a standard measure in language modeling, quantifying how well a probability model predicts a sample. It's calculated as the exponential of the average negative log-likelihood of the evaluation dataset according to the model. Mathematically, for a sequence of tokens $W = w_1, w_2, ..., w_N$ , perplexity (PPL) is:

PPL(W) = \exp\left( -\frac{1}{N} \sum_{i=1}^{N} \log p(w_i | w_1, ..., w_{i-1}) \right)

A lower perplexity score generally indicates that the model is more confident and accurate in predicting the sequence of tokens in the evaluation set. In the context of SFT evaluation, you typically compute perplexity on the response part of the prompt-response pairs in your held-out dataset.

While a decreasing perplexity on the validation set during training suggests the model is learning the data distribution, it's not a direct measure of quality or instruction following. A model might achieve low perplexity by generating repetitive or generic text that matches the statistics of the SFT data but fails to produce helpful or specific responses. However, a significantly high or increasing validation perplexity is often a red flag indicating training problems or severe overfitting.

Sequence-Overlap Metrics (ROUGE, BLEU)

Metrics like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BLEU (Bilingual Evaluation Understudy) measure the overlap between the generated response and one or more reference responses.

ROUGE: Often used in summarization, ROUGE variants (ROUGE-1, ROUGE-2, ROUGE-L) measure overlap of unigrams, bigrams, and the longest common subsequence, respectively.
BLEU: Commonly used in machine translation, BLEU measures n-gram precision, penalizing generations that are too short.

These metrics can be useful if your SFT task involves generating text that should closely match a specific reference (e.g., closed-ended question answering, specific formatting tasks). However, for more open-ended generation tasks where multiple valid and diverse responses are possible, ROUGE and BLEU scores can be misleading. High overlap doesn't necessarily mean high quality, and low overlap doesn't necessarily mean poor quality. Use them cautiously, primarily as sanity checks or for tasks where strong reference overlap is expected.

Loss Curves Analysis

Monitoring training and validation loss curves is fundamental. Ideally, both losses should decrease and stabilize. A large gap between training and validation loss indicates overfitting, where the model has memorized the training data but doesn't generalize well. Spikes or instability in the validation loss might point to issues with hyperparameters like the learning rate or batch size.

Typical SFT loss curves showing decreasing training and validation loss, with a small gap indicating reasonable generalization.

Model-Based Evaluation

An increasingly common technique involves using another powerful LLM (e.g., GPT-4, Claude) as an automated evaluator. The process typically looks like this:

Provide a prompt from the evaluation set to the SFT model to generate a response.
Provide the same prompt, the SFT model's response, and potentially a reference response to the evaluator LLM.
Instruct the evaluator LLM to assess the SFT response based on specific criteria (e.g., "Rate the helpfulness of this response on a scale of 1-5," "Did the response follow all instructions in the prompt?", "Is the format correct?").

This approach offers better scalability than human evaluation and can assess aspects beyond simple text overlap. However, it relies on the capabilities and potential biases of the evaluator model, and results can sometimes be inconsistent or require careful prompt engineering for the evaluator itself.

Human Evaluation

Despite the challenges of cost and scalability, direct human evaluation remains the gold standard for assessing the nuanced aspects of LLM behavior targeted by SFT.

Qualitative Assessment

This involves human reviewers examining a sample of model outputs generated from evaluation prompts. Reviewers look for:

Instruction Adherence: Does the model follow the specific constraints and requirements of the prompt?
Style and Tone: Does the output match the desired style (e.g., formal, conversational, code-like) demonstrated in the SFT data?
Factual Accuracy: (If applicable) Are the model's claims correct?
Helpfulness and Relevance: Is the response useful and directly addressing the prompt?
Safety and Harmlessness: Does the model avoid generating harmful, biased, or inappropriate content?
Failure Modes: Are there signs of repetition, hallucination, incoherence, or overly generic responses?

Reviewers often provide written feedback and examples of good and bad outputs.

Quantitative Assessment

Human evaluation can also be structured to yield quantitative scores:

Likert Scales: Raters score responses on predefined scales (e.g., 1-5) for dimensions like helpfulness, clarity, adherence to format, etc.
Pairwise Comparison: Raters are shown a prompt and two responses (e.g., one from the SFT model, one from the base model, or two variants from the SFT model) and asked to choose which one is better, potentially explaining why. This forms the basis for preference data used later in reward modeling but can also be used here to compare the SFT model against its baseline.

Clear annotation guidelines and rater training are important for ensuring consistency and reliability in human evaluation results.

Overview of different approaches for evaluating the SFT model using a held-out evaluation dataset.

Interpreting the Results

The goal of SFT evaluation is not necessarily to achieve state-of-the-art scores on general benchmarks, but to confirm readiness for the next RLHF stage. Look for:

Clear Improvement: The SFT model should demonstrably follow instructions and adopt the target style better than the base pre-trained model, especially confirmed by human evaluation.
No Catastrophic Forgetting: The model should retain its general language capabilities and not suffer severe regressions in fluency or coherence.
Acceptable Failure Modes: While perfect performance is unlikely, any failure modes (like occasional repetition or minor format errors) should be understood and deemed acceptable as a starting point for RL tuning. Severe issues might necessitate revisiting the SFT dataset curation or training process.

Ultimately, evaluating the SFT model is about building confidence. A well-performing SFT model, confirmed through a combination of automated metrics and, ideally, human review, provides a much stronger starting policy for the PPO fine-tuning stage. This allows the RL process to focus more effectively on optimizing for nuanced human preferences captured by the reward model, rather than struggling to learn basic instruction following or stylistic conformance from scratch.