Logging Metrics during Training/Evaluation

While TensorBoard provides rich visualizations, the foundation of effective monitoring is systematic logging within your training and evaluation code. Simply running the loops isn't enough; you need to record important performance indicators to understand how the training is progressing and to diagnose issues when they arise. Here is how to implement basic metric logging directly within your PyTorch training and evaluation routines.

Why Log Metrics?

Logging metrics serves several important purposes:

1. Performance Tracking: Observe trends in loss and accuracy (or other relevant metrics) over epochs to see if the model is learning effectively.
2. Debugging: Identify potential problems early. Is the loss decreasing? Is it stagnating? Is the validation performance improving or degrading? Anomalies in logged metrics often point to underlying issues discussed earlier, like learning rate problems or overfitting.
3. Comparison: Logged metrics allow for objective comparison between different model architectures, hyperparameters, or training runs.
4. Foundation for Visualization: Tools like TensorBoard rely on these logged values to generate plots and dashboards.

Metrics to Log

The specific metrics depend on your task, but some common ones include:

• Loss: This is the value your optimizer is trying to minimize. You should almost always log the loss. It's essential to track both the training loss (calculated on the training data within the training loop) and the validation loss (calculated on a separate validation dataset within the evaluation loop). Comparing these two is critical for identifying overfitting.
• Accuracy: For classification tasks, accuracy (the proportion of correctly classified examples) is a standard, interpretable metric.
• Other Task-Specific Metrics: Depending on the problem, you might log Precision, Recall, F1-score (for classification), Mean Absolute Error (MAE) or Root Mean Squared Error (RMSE) (for regression), Intersection over Union (IoU) (for segmentation), etc.

Implementing Logging in the Training Loop

During training, you typically want to track the average loss and accuracy over each epoch. Logging metrics for every single batch can be noisy and less informative about overall trends, although it can sometimes be useful for debugging instability.

Here’s how you might modify a standard training epoch function to include logging:

import torch

# Assume model, train_dataloader, loss_fn, optimizer are defined

def train_one_epoch(model, train_dataloader, loss_fn, optimizer, device):
    model.train() # Set model to training mode
    running_loss = 0.0
    correct_predictions = 0
    total_samples = 0

    for batch_idx, (inputs, labels) in enumerate(train_dataloader):
        inputs, labels = inputs.to(device), labels.to(device)

        # 1. Zero gradients
        optimizer.zero_grad()

        # 2. Forward pass
        outputs = model(inputs)

        # 3. Calculate loss
        loss = loss_fn(outputs, labels)

        # 4. Backward pass
        loss.backward()

        # 5. Optimizer step
        optimizer.step()

        # --- Logging steps ---
        # Accumulate loss (use .item() to get Python number)
        running_loss += loss.item() * inputs.size(0) # Weighted by batch size

        # Accumulate accuracy (example for classification)
        _, predicted = torch.max(outputs.data, 1)
        total_samples += labels.size(0)
        correct_predictions += (predicted == labels).sum().item()
        # --- End Logging steps ---

    # Calculate average loss and accuracy for the epoch
    epoch_loss = running_loss / total_samples
    epoch_acc = correct_predictions / total_samples

    print(f"Training Epoch: Loss: {epoch_loss:.4f}, Accuracy: {epoch_acc:.4f}")

    # Return metrics for potential further logging or analysis
    return epoch_loss, epoch_acc

Important points in the logging implementation:

• Initialize accumulators (running_loss, correct_predictions, total_samples) at the start of the epoch.
• Inside the batch loop, after calculating the loss and predictions, update the accumulators.
  • Use loss.item() to get the scalar value of the loss tensor for the current batch, preventing graph retention. Multiply by inputs.size(0) (batch size) if you want the total loss before averaging at the end; otherwise, you can average batch losses, but weighting by batch size is more accurate if the last batch is smaller.
  • Calculate accuracy (or other metrics) for the batch and add to the running totals. Use .item() here as well.
• After the loop finishes, calculate the average loss and accuracy for the entire epoch by dividing the accumulated totals by the number of samples.
• Print or store these epoch-level metrics.

Implementing Logging in the Evaluation Loop

Logging in the evaluation loop is similar, but important differences exist:

• It runs under torch.no_grad() to disable gradient calculations.
• The model should be in evaluation mode (model.eval()) to disable dropout and use batch normalization statistics learned during training.
• There's no backpropagation or optimizer step.

import torch

# Assume model, val_dataloader, loss_fn are defined

def evaluate_model(model, val_dataloader, loss_fn, device):
    model.eval() # Set model to evaluation mode
    running_loss = 0.0
    correct_predictions = 0
    total_samples = 0

    with torch.no_grad(): # Disable gradient computations
        for inputs, labels in val_dataloader:
            inputs, labels = inputs.to(device), labels.to(device)

            # Forward pass
            outputs = model(inputs)

            # Calculate loss
            loss = loss_fn(outputs, labels)

            # --- Logging steps ---
            running_loss += loss.item() * inputs.size(0)

            _, predicted = torch.max(outputs.data, 1)
            total_samples += labels.size(0)
            correct_predictions += (predicted == labels).sum().item()
            # --- End Logging steps ---

    epoch_loss = running_loss / total_samples
    epoch_acc = correct_predictions / total_samples

    print(f"Validation: Loss: {epoch_loss:.4f}, Accuracy: {epoch_acc:.4f}")
    return epoch_loss, epoch_acc

Storing and Using Logged Metrics

Printing metrics to the console is useful for immediate feedback. For more systematic analysis or visualization, you'll want to store them. Simple Python lists or dictionaries work well:

# --- Inside your main training script ---
num_epochs = 10
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# ... initialize model, data, loss, optimizer ...

train_losses, train_accuracies = [], []
val_losses, val_accuracies = [], []

for epoch in range(num_epochs):
    print(f"--- Epoch {epoch+1}/{num_epochs} ---")
    train_loss, train_acc = train_one_epoch(model, train_dataloader, loss_fn, optimizer, device)
    val_loss, val_acc = evaluate_model(model, val_dataloader, loss_fn, device)

    # Store metrics
    train_losses.append(train_loss)
    train_accuracies.append(train_acc)
    val_losses.append(val_loss)
    val_accuracies.append(val_acc)

    # Optional: Save model checkpoint here based on validation performance
    # Optional: Log metrics to TensorBoard (using values stored above)
    # writer.add_scalar('Loss/train', train_loss, epoch)
    # writer.add_scalar('Loss/validation', val_loss, epoch)
    # ... etc.

print("Training finished.")
# Now you can analyze the lists: train_losses, val_losses, etc.
# For example, save them to a file or plot them.

This structure allows you to collect performance data over the entire training process. These stored lists (train_losses, val_losses, etc.) are exactly what you would feed into plotting libraries like Matplotlib or pass to a TensorBoard SummaryWriter (as discussed in the previous section) to create visualizations like the one below.

Training and validation loss curves plotted over 10 epochs. Observing these trends helps diagnose overfitting (validation loss increasing while training loss decreases) or underfitting (both losses remain high).

By consistently logging metrics during both training and evaluation, you gain essential visibility into your model's behavior, enabling informed decisions about hyperparameter tuning, model adjustments, and debugging strategies.