Evaluating the performance of models adapted using Parameter-Efficient Fine-Tuning (PEFT) techniques requires selecting appropriate metrics that reflect success on the intended downstream tasks. As outlined in the chapter introduction, simply training a model isn't enough; we need objective measures to understand how well methods like LoRA, QLoRA, or Adapter Tuning perform compared to each other and to traditional full fine-tuning. The choice of metric is heavily dependent on the specific application, whether it involves understanding language (NLU) or generating it (NLG).

Metrics for Natural Language Understanding (NLU) Tasks

NLU tasks typically involve classification, sequence labeling, or question answering. PEFT methods are often evaluated on established benchmarks like GLUE (General Language Understanding Evaluation) or SuperGLUE, which encompass a variety of these tasks.

Classification Tasks

For tasks like sentiment analysis, topic classification, or natural language inference, where the goal is to assign a label to a given input text, standard classification metrics apply:

Accuracy: The simplest metric, representing the proportion of correct predictions. While intuitive, it can be misleading on imbalanced datasets. $\text{Accuracy} = \frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}}$
Precision, Recall, and F1-Score: These metrics provide a more detailed view, especially for imbalanced classes.
- Precision: Measures the accuracy of positive predictions. $\text{Precision} = \frac{TP}{TP + FP}$ (where TP = True Positives, FP = False Positives).
- Recall (Sensitivity): Measures the proportion of actual positives that were correctly identified. $\text{Recall} = \frac{TP}{TP + FN}$ (where FN = False Negatives).
- F1-Score: The harmonic mean of Precision and Recall, providing a single score that balances both. $\text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$ . This is often a primary metric for classification tasks in benchmarks.
Matthews Correlation Coefficient (MCC): Considered a balanced measure even for imbalanced classes, ranging from -1 (total disagreement) to +1 (perfect prediction). $\text{MCC} = \frac{TP \times TN - FP \times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$ (where TN = True Negatives).

When evaluating PEFT, we use these metrics to compare the performance achieved with, for example, a LoRA adapter against the performance of the fully fine-tuned base model. The goal is often to achieve performance close to full fine-tuning (e.g., within 1-2% F1 points) while using significantly fewer trainable parameters.

Question Answering (QA) Tasks

For extractive QA tasks (like SQuAD - Stanford Question Answering Dataset), where the answer is a span of text within a given context, common metrics include:

Exact Match (EM): Measures the percentage of predictions that match the ground truth answer exactly. It's a strict metric.
F1-Score: Calculated at the token level, treating prediction and ground truth as bags of tokens. It measures the overlap between the predicted and ground truth answer spans, offering partial credit for partially correct answers. This is often considered a more robust metric than EM.

Metrics for Natural Language Generation (NLG) Tasks

Evaluating generated text (e.g., summarization, translation, dialogue) is inherently more complex than evaluating NLU tasks because multiple valid outputs can exist. Metrics typically rely on comparing the generated text to one or more reference texts.

BLEU (Bilingual Evaluation Understudy): Primarily used in machine translation, BLEU measures n-gram precision overlap between the generated text and reference translations. It penalizes sentences that are too short. Higher scores indicate better similarity to references.
ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Commonly used for summarization, ROUGE measures n-gram recall overlap. Variants include:
- ROUGE-N: Measures overlap of n-grams (e.g., ROUGE-1 for unigrams, ROUGE-2 for bigrams).
- ROUGE-L: Measures the longest common subsequence (LCS) between the generated and reference summaries, capturing sentence-level structure similarity.
METEOR (Metric for Evaluation of Translation with Explicit ORdering): Also used in translation and generation, METEOR considers exact matches, stemmed matches, synonym matches, and paraphrases, aligning prediction and reference based on these criteria. It includes a penalty for incorrect word order.
Perplexity (PPL): An intrinsic evaluation metric measuring how well a probability model predicts a sample. Lower perplexity indicates the model is less surprised by the test data, suggesting better language modeling capabilities. While useful during training, it doesn't always correlate perfectly with human judgments of quality on downstream tasks.

Comparing different PEFT methods against full fine-tuning on a hypothetical NLU task using the F1 score. PEFT methods often approach the performance of full fine-tuning.

Considerations for PEFT Evaluation

Beyond the standard metrics for specific tasks, evaluating PEFT involves additional considerations:

Performance vs. Parameter Count: A primary goal of PEFT is efficiency. Evaluation should always consider the performance achieved relative to the number of trainable parameters. A method might achieve slightly lower performance on standard metrics but be vastly more efficient, making it preferable in resource-constrained environments.
Sensitivity to Hyperparameters: PEFT methods like LoRA have specific hyperparameters (e.g., rank $r$ , scaling factor $\alpha$ ). Evaluation should ideally explore the sensitivity of performance metrics to these settings.
Task Transferability: How well does a PEFT module trained on one task perform on a closely related task? Evaluating transferability can provide insights into the generalization capabilities of the learned adaptations.
Human Evaluation: For NLG tasks, automated metrics often fall short of capturing aspects like fluency, coherence, and creativity. Human evaluation, although expensive and time-consuming, remains an important component for a comprehensive assessment, especially when subtle quality differences are expected.

Selecting and interpreting the right set of metrics is fundamental to understanding the trade-offs inherent in using PEFT techniques. It allows for informed decisions about which method and configuration best suit the specific task requirements and operational constraints. The subsequent sections will delve deeper into benchmarking, analyzing robustness, and assessing the computational costs associated with these methods.