Common Loss Functions for Regression (MSE, MAE)

Training a neural network involves minimizing the difference between its predictions and the actual target values. For regression problems, which aim to predict a continuous numerical value (such as house prices or stock values), specific loss functions are crucial. Two primary examples for this task are Mean Squared Error (MSE) and Mean Absolute Error (MAE).

Mean Squared Error (MSE)

Mean Squared Error, often called MSE or L2 loss, is perhaps the most widely used loss function for regression. It measures the average of the squares of the errors between the predicted values and the true values.

Mathematically, for $n$ data points, where $y_i$ is the true target value and $\hat{y}_i$ is the predicted value for the $i$-th data point, MSE is defined as:

$$L_{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

Why square the difference?

1. Non-negativity: Squaring ensures the error term is always non-negative. A perfect prediction yields an error of 0.
2. Penalizing Large Errors: The squaring operation heavily penalizes larger prediction errors more than smaller ones. For instance, an error of 4 contributes 16 to the sum, while an error of 2 contributes only 4. This characteristic makes MSE sensitive to outliers, as a single large error can significantly inflate the total loss.
3. Mathematical Convenience: The MSE function is continuously differentiable, resulting in a smooth gradient. This is beneficial for gradient-based optimization algorithms like gradient descent, as it provides a clear direction for parameter updates. The gradient points towards the steepest increase in the loss, so moving in the opposite direction helps minimize it.

The units of MSE are the square of the units of the target variable (e.g., dollars squared if predicting prices). While this might seem unintuitive, the Root Mean Squared Error (RMSE), simply $\sqrt{MSE}$, is often reported for interpretability as it has the same units as the target variable. However, MSE is typically minimized during training due to its favorable mathematical properties.

Mean Absolute Error (MAE)

Mean Absolute Error, also known as MAE or L1 loss, offers an alternative perspective on measuring regression error. Instead of squaring the differences, MAE calculates the average of the absolute differences between predictions and true values.

The formula for MAE is:

$$L_{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$$

Characteristics of MAE:

1. Robustness to Outliers: Because MAE uses the absolute difference, it doesn't disproportionately penalize large errors as MSE does. An error of 4 contributes 4 to the sum, and an error of 2 contributes 2. This linear penalty makes MAE less sensitive to outliers in the dataset compared to MSE. If your data contains significant outliers that you don't want to dominate the loss calculation, MAE might be a better choice.
2. Interpretability: The resulting MAE value is in the same units as the target variable (e.g., dollars if predicting prices), making it directly interpretable as the average absolute deviation from the true values.
3. Gradient Behavior: The MAE function has a constant gradient magnitude ($+1$ or $-1$) for non-zero errors. While this prevents vanishing gradients for large errors (unlike MSE where the gradient approaches zero as error approaches zero), it can lead to less stable convergence near the minimum, as the gradient doesn't shrink. Also, the derivative is undefined exactly at zero error, though this is usually handled in practice by using a sub-gradient or skipping updates when the error is exactly zero.

Comparing MSE and MAE

The choice between MSE and MAE depends on the specific problem and data characteristics.

• Sensitivity to Outliers: MSE penalizes outliers heavily; MAE is more direct.
• Gradient: MSE has a smooth gradient that decreases as the error gets smaller, potentially leading to more stable convergence near the minimum. MAE has a constant gradient magnitude, which can be beneficial for large errors but might require adjusting the learning rate as training progresses.
• Interpretability: MAE's units match the target variable, making it easier to interpret directly.

Here's a visual comparison of how MSE and MAE penalize errors:

Comparison of MSE (blue) and MAE (green) loss values based on the prediction error. Note the quadratic increase for MSE versus the linear increase for MAE.

In practice, MSE is often the default choice due to its mathematical properties aligning well with gradient descent. However, if outliers are a significant concern, experimenting with MAE (or other robust loss functions like Huber loss, which combines aspects of both) is worthwhile.

Here's a quick example using PyTorch to calculate both losses:

import torch
import torch.nn as nn

# Sample predictions and targets
predictions = torch.tensor([2.5, 0.0, 1.8, 9.1])
targets = torch.tensor([3.0, -0.5, 2.0, 8.0])

# Calculate MSE Loss
mse_loss_fn = nn.MSELoss()
mse_loss = mse_loss_fn(predictions, targets)

# Calculate MAE Loss (L1Loss in PyTorch)
mae_loss_fn = nn.L1Loss()
mae_loss = mae_loss_fn(predictions, targets)

print(f"Predictions: {predictions.numpy()}")
print(f"Targets:     {targets.numpy()}")
print(f"Errors:      {(targets - predictions).numpy()}")
print(f"Squared Errors: {((targets - predictions)**2).numpy()}")
print(f"Absolute Errors: {torch.abs(targets - predictions).numpy()}")
print("-" * 30)
print(f"MSE Loss: {mse_loss.item():.4f}") # Average of [0.25, 0.25, 0.04, 1.21] = 1.75 / 4 = 0.4375
print(f"MAE Loss: {mae_loss.item():.4f}") # Average of [0.5, 0.5, 0.2, 1.1] = 2.3 / 4 = 0.5750

This simple calculation demonstrates how the squaring in MSE gives more weight to the larger error (1.1) compared to MAE. Understanding these differences is important when selecting the appropriate loss function to guide your model's training process towards making accurate predictions.