Fine-tuning Procedures for Evaluation

To effectively evaluate a pre-trained Large Language Model (LLM) on downstream tasks, we often need to adapt it slightly to the specific format and objective of the target task. This process is known as fine-tuning. While zero-shot or few-shot evaluations (discussed later) test the model's inherent generalization capabilities, fine-tuning allows the model to learn task-specific patterns from labeled data, often yielding higher performance and providing a clearer picture of the pre-trained model's potential utility for that task.

Fine-tuning uses the powerful representations learned during pre-training and adjusts them using a relatively small amount of task-specific labeled data. The core idea is that the pre-trained model already understands language structure, semantics, and context; fine-tuning simply teaches it how to apply this knowledge to a new problem formulation.

The General Fine-tuning Workflow

The standard procedure for fine-tuning an LLM for evaluation on a downstream task involves several steps:

1. Load Pre-trained Model: Start with the weights of a pre-trained LLM. This could be a model you trained yourself or a publicly available checkpoint (e.g., from Hugging Face Hub).
2. Adapt Model Architecture: Modify the model by adding a task-specific "head." This head is typically a small neural network layer (or layers) placed on top of the pre-trained model's core structure (e.g., the Transformer blocks). The nature of this head depends entirely on the downstream task.
3. Prepare Task Dataset: Obtain the labeled dataset for the downstream task. This involves formatting the input data according to the requirements of the model and the task head, and tokenizing the text using the same tokenizer used during the model's pre-training.
4. Define Objective: Choose a loss function appropriate for the task (e.g., Cross-Entropy Loss for classification, Mean Squared Error for regression).
5. Train: Run the training loop, updating the weights of the entire model or just the newly added head and potentially the top layers of the pre-trained model. Use a suitable optimizer (like AdamW) and a learning rate schedule. Training typically runs for only a few epochs on the task dataset.
6. Evaluate: Measure performance on a held-out test set using metrics relevant to the downstream task (e.g., Accuracy, F1 score, ROUGE, BLEU).

Here's a diagram of the process:

A pre-trained LLM's core layers provide input representations to a newly added task-specific head. The head makes predictions, which are compared against labels from the task dataset using a task-specific loss function. Gradients from this loss update the weights of the head and typically some or all of the pre-trained layers.

Task-Specific Heads and Data Formatting

The adaptation step primarily involves adding the correct head and formatting the data appropriately. Let's look at common examples:

Text Classification

• Goal: Assign a category label to a piece of text (e.g., sentiment analysis, topic classification).
• Head: Often a single linear layer that takes the final hidden state of a designated token (like the [CLS] token in BERT-style models, or the last token in causal models) and projects it to the number of output classes. A dropout layer is commonly added before the linear layer for regularization.
• Input Format: For models like BERT, the input is typically formatted as [CLS] text_sequence [SEP]. For causal models (like GPT), the input might just be text_sequence, and the representation of the final token is used.
• Loss: Cross-Entropy Loss is standard for multi-class classification.

import torch
import torch.nn as nn
from transformers import ( # Example using Hugging Face Transformers
    AutoModel, AutoTokenizer
)

# Load pre-trained model (e.g., BERT)
model_name = "bert-base-uncased"
pretrained_model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Define classification head
num_labels = 3 # Example: Positive, Negative, Neutral sentiment
hidden_size = pretrained_model.config.hidden_size
classification_head = nn.Sequential(
    nn.Dropout(0.1),
    nn.Linear(hidden_size, num_labels)
)

# Example input processing
text = "This is an example sentence."
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)

# Forward pass (simplified)
# Get hidden states from the pre-trained model
outputs = pretrained_model(**inputs)
# Use the representation of the [CLS] token (first token)
cls_representation = outputs.last_hidden_state[:, 0, :]
# Pass through the classification head
logits = classification_head(cls_representation)

# 'logits' now contains raw scores for each class
# Apply CrossEntropyLoss using these logits and target labels during training

Extractive Question Answering (QA)

• Goal: Find the span of text within a given context passage that answers a specific question (e.g., SQuAD benchmark).
• Head: Typically consists of two linear layers. One predicts the probability of each token in the context being the start of the answer span, and the other predicts the probability of each token being the end of the answer span.
• Input Format: Usually combines the question and context, separated by a special token: [CLS] question_text [SEP] context_text [SEP].
• Loss: Cross-Entropy Loss is calculated separately for the start and end positions. The model is trained to predict the correct start and end token indices within the context. During inference, post-processing is needed to find the most likely valid span (where end index >= start index).

# (Continuing from previous imports)

# Define QA head
qa_head = nn.Linear(hidden_size, 2)
# Output: start_logit, end_logit for each token

# Example input processing
question = "What is the capital of Malaysia?"
context = "Malaysia is a Southeast Asian country. Kuala Lumpur is its capital and largest city."
inputs = tokenizer(
    question,
    context,
    return_tensors="pt",
    padding=True,
    truncation=True
)

# Forward pass (simplified)
outputs = pretrained_model(**inputs)
sequence_output = outputs.last_hidden_state
# Shape: (batch_size, seq_len, hidden_size)

# Pass sequence output through the QA head
logits = qa_head(sequence_output) # Shape: (batch_size, seq_len, 2)
start_logits, end_logits = logits.split(1, dim=-1)
start_logits = start_logits.squeeze(-1) # Shape: (batch_size, seq_len)
end_logits = end_logits.squeeze(-1)    # Shape: (batch_size, seq_len)

# 'start_logits' and 'end_logits' contain scores for start/end positions
# Apply CrossEntropyLoss using these and target start/end indices
# during training

Sequence-to-Sequence Tasks

• Goal: Generate a sequence of text based on an input sequence (e.g., summarization, translation, dialogue generation).
• Head: For encoder-decoder models (like T5, BART), the pre-trained decoder already serves as the generation head. For decoder-only models (like GPT), the standard language modeling head (predicting the next token) is used directly for generation. Fine-tuning adjusts the model to generate outputs conditioned on the specific input format (e.g., adding a prefix like "summarize: " to the input text).
• Input Format: Varies based on the task. For summarization: input_article. For translation: translate English to French: input_english_sentence. The target sequence is used as labels during training.
• Loss: Cross-Entropy Loss computed over the generated sequence tokens compared to the target sequence (often using teacher forcing during training, where the model sees the ground truth previous token).

Training for Fine-tuning

Fine-tuning involves training, but with some differences compared to pre-training:

• Learning Rate: Significantly smaller learning rates (e.g., $1e-5$ to $5e-5$) are typically used. The pre-trained model's weights are already well-initialized, so large updates can disrupt the learned representations.
• Optimizer: AdamW remains a common choice due to its effectiveness and handling of weight decay.
• Batch Size: Often constrained by GPU memory, especially with larger models. Effective batch size can sometimes be increased using gradient accumulation.
• Epochs: Fine-tuning usually requires only a small number of epochs (often 1-5). Training for too long on the smaller, specific dataset can lead to overfitting and degrade the model's general capabilities.
• Learning Rate Schedule: A linear decay schedule with a brief warmup period (e.g., warming up over the first 6-10% of training steps) is often effective.
• Weight Decay: Applied as a regularization technique, similar to pre-training.
• Freezing Layers: Sometimes, only the task-specific head and the top few layers of the pre-trained model are trained initially, keeping the lower layers frozen. This can save computation and prevent catastrophic forgetting, though full fine-tuning (updating all parameters) often yields the best performance if sufficient data is available.

Simplified Fine-tuning Loop Example (PyTorch)

import torch
from torch.optim import AdamW
from torch.utils.data import DataLoader, Dataset
from transformers import get_linear_schedule_with_warmup

# Assume 'model' includes the pre-trained backbone and the task-specific head
# Assume 'train_dataset' is a PyTorch Dataset yielding formatted,
# tokenized inputs and labels
# Assume 'loss_fn' is the appropriate loss function (e.g., nn.CrossEntropyLoss)

learning_rate = 3e-5
num_epochs = 3
batch_size = 16
warmup_steps = 100
total_training_steps = len(train_dataset) * num_epochs // batch_size # Approximation

optimizer = AdamW(model.parameters(), lr=learning_rate)
scheduler = get_linear_schedule_with_warmup(optimizer,
                                           num_warmup_steps=warmup_steps,
                                           num_training_steps=total_training_steps)
train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.train()

for epoch in range(num_epochs):
    for batch in train_dataloader:
        # Move batch to device
        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        labels = batch['labels'].to(device) # Assuming labels are part of the batch

        # Clear previous gradients
        optimizer.zero_grad()

        # Forward pass - Get model predictions (logits)
        # The exact way to get logits depends on the model and head structure
        # For classification: logits = model(
        #     input_ids=input_ids, attention_mask=attention_mask).logits
        # For QA: outputs = model(...); start_logits, end_logits =
        #     outputs.start_logits, outputs.end_logits
        # This needs to be adapted based on the specific model wrapper and task
        outputs = model(input_ids=input_ids,
                        attention_mask=attention_mask)
        logits = outputs.logits # Adjust based on actual model output structure

        # Calculate loss
        # Loss calculation depends on the task (e.g., shape of logits vs labels)
        # For classification: loss = loss_fn(
        #     logits.view(-1, num_labels), labels.view(-1))
        # For QA: loss = (loss_fn(start_logits, start_positions) +
        #                  loss_fn(end_logits, end_positions)) / 2
        loss = loss_fn(logits, labels) # Adjust based on specific task loss needs

        # Backward pass
        loss.backward()

        # Update weights
        optimizer.step()
        scheduler.step()

    print(f"Epoch {epoch + 1} completed. Last batch loss: {loss.item()}")

# After training, evaluate on the test set using task-specific metrics

Evaluation Metrics

Crucially, after fine-tuning, the model is evaluated using metrics specific to the downstream task. For classification, this might be accuracy or F1-score. For QA, Exact Match (EM) and F1-score over the predicted answer tokens are common. For summarization, ROUGE scores (ROUGE-1, ROUGE-2, ROUGE-L) are standard, measuring overlap with reference summaries. For translation, BLEU score is often used. These extrinsic metrics provide a direct measure of the model's performance on the task it was adapted for, complementing the insights from intrinsic metrics like perplexity.

Fine-tuning is a powerful technique for evaluating and adapting LLMs. While it requires labeled data and computational resources (though far less than pre-training), it allows us to assess how effectively a pre-trained model's learned knowledge can be transferred to solve specific, practical problems. The results obtained through fine-tuning often represent a strong performance baseline for the underlying pre-trained model on that particular task.