Defining Performance Metrics for Generative Tasks

When evaluating a fine-tuned model, a fundamental question arises: Is it any good? For generative tasks, answering this is not as straightforward as checking an accuracy score. Unlike a classification problem where an output is either right or wrong, the quality of generated text is subjective and multifaceted. A summary can be factually correct but poorly written. A translation can be literal but awkward. A chatbot response can be fluent but unhelpful.

Therefore, evaluating a fine-tuned generative model requires a thoughtful approach tailored to its intended function. The metrics you choose must reflect the specific goals of your fine-tuning task. A model trained to summarize legal documents will be judged by different standards than one designed to generate creative marketing copy.

The Spectrum of Generative Tasks

The first step in defining an evaluation strategy is to identify the primary function of your model. Different tasks place emphasis on different aspects of text quality.

Summarization

The goal is to produce a concise and accurate representation of a longer text. Evaluation must measure:

• Fidelity: Does the summary accurately reflect the important information from the source text without introducing falsehoods?
• Conciseness: Is the summary significantly shorter than the original?
• Fluency: Is the summary grammatically correct and easy to read?

Automated metrics for this task often rely on word or phrase overlap with a human-written "reference" summary.

Machine Translation

Here, the objective is to convey the exact meaning of a source text in a different language. The evaluation focuses on:

• Accuracy: Is the meaning of the original text preserved?
• Fluency: Does the output read like a natural sentence in the target language?

Similar to summarization, evaluation often involves comparing the model's output to one or more professional human translations.

Instruction Following and Question Answering

For these tasks, the model must provide a direct and correct response to a user's prompt. The criteria for success are:

• Relevance: Does the response directly address the user's instruction or question?
• Correctness: Is the information provided factually accurate?
• Clarity: Is the answer presented in a clear and understandable way?

Factual correctness is notoriously difficult to measure automatically and often requires human review or comparison against a known knowledge base.

Conversational AI

Evaluating a chatbot or conversational agent is complex because it involves multi-turn interactions. A single good response is not enough; the entire conversation must be effective. Important evaluation points include:

• Coherence: Do the model's responses logically follow the conversation's history?
• Helpfulness: Does the model successfully assist the user with their goal?
• Safety: Does the model avoid generating harmful, biased, or inappropriate content?
• Engagement: Is the conversation natural and engaging for the user?

Two Pillars of Evaluation: Automated and Human

Given the diverse requirements of generative tasks, an evaluation strategy rests on two distinct but complementary approaches: automated quantitative metrics and human-led qualitative assessments.

An evaluation framework combines fast, automated metrics for continuous monitoring with slower, more thorough human assessments to measure true performance.

Automated Metrics

These are algorithms that compute a score by comparing the model's output to a reference text. Their primary advantages are speed and scalability. You can run them automatically on thousands of examples to get a consistent measure of performance as you iterate on your model. They are indispensable for tracking progress during training and for comparing different model versions.

Common automated metrics fall into a few categories:

• Overlap-based: These metrics, such as ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BLEU (Bilingual Evaluation Understudy), measure the overlap of n-grams (sequences of words) between the generated text and a reference text. They are standards for summarization and translation.
• Model-based: Metrics like Perplexity measure the model's uncertainty in predicting the next token. It is derived from the cross-entropy loss and can be a useful proxy for fluency. A lower perplexity, often calculated as $2^{H(p,q)}$ where $H(p,q)$ is the cross-entropy loss, indicates the model is more "confident" about the text it is generating.

We will cover the implementation of these metrics in the next section.

Human Evaluation

No automated metric can perfectly capture the quality of generated text. Does the output sound natural? Is it genuinely creative? Is it factually correct? Answering these questions requires human judgment. Human evaluation is the gold standard for assessing model performance, though it is more time-consuming and costly.

Common methods for human evaluation include:

• Likert Scales: Raters score model outputs on a scale (e.g., 1 to 5) for specific attributes like fluency, relevance, and coherence.
• A/B Testing: A rater is shown two outputs from different models (or a model and a baseline) and asked to choose the better one.
• Direct Assessment: Raters are asked to perform tasks like fact-checking the output against a source or editing it to make it perfect.

A comprehensive evaluation strategy uses automated metrics for rapid, iterative feedback and periodic human evaluation to ensure the model is truly meeting its goals. With this framework in place, we can now examine the implementation details of the most common quantitative metrics.