Before a neural network can learn, we need a way to precisely measure how well it's performing on a given task. Is it predicting house prices accurately? Is it correctly classifying images of cats and dogs? This measurement is the role of the loss function, also sometimes called a cost function or objective function.

Think of the loss function as grading the network's predictions. It takes the network's output (the predictions, often denoted as $\hat{y}$ ) and compares them to the true target values ( $y$ ) from our dataset. The result is a single scalar value representing the "error" or "loss" for those predictions. A perfect prediction would result in zero loss, while larger errors lead to higher loss values.

The goal of training is to minimize this loss value. The entire machinery of gradient descent and backpropagation, which we'll discuss next, is geared towards adjusting the network's weights and biases in a way that systematically reduces the output of the chosen loss function.

The specific loss function you choose is significant and depends heavily on the type of problem you're trying to solve, primarily whether it's a regression or a classification task.

Loss Functions for Regression Problems

Regression problems involve predicting continuous numerical values, like the price of a house, the temperature tomorrow, or the stock market value.

Mean Squared Error (MSE)

The most common loss function for regression is the Mean Squared Error (MSE). It calculates the average of the squared differences between the predicted values and the actual values.

For a dataset with $N$ examples, the MSE is calculated as:

L_{MSE} = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2

Where:

$N$ is the number of data points (e.g., in a batch).
$y_i$ is the true target value for the $i$ -th data point.
$\hat{y}_i$ is the predicted value by the network for the $i$ -th data point.

Why square the difference?

Direction doesn't matter: Squaring ensures that errors (differences) are always positive. An under-prediction ( $y_i > \hat{y}_i$ ) is penalized just like an over-prediction ( $y_i < \hat{y}_i$ ).
Penalizes larger errors more: The squaring operation means that larger differences contribute much more to the total loss than smaller differences. For example, an error of 2 contributes 4 to the sum, while an error of 4 contributes 16. This can be desirable if large errors are particularly problematic for your application.
Mathematical convenience: The MSE function is smooth and differentiable, which makes it easy to calculate gradients for backpropagation.

Consideration: Because MSE squares the error term, it can be sensitive to outliers. A single data point with a very large error can dominate the loss value and potentially skew the training process.

The relationship between prediction error and the contribution to MSE loss. Larger errors (positive or negative) result in quadratically increasing loss values.

Mean Absolute Error (MAE)

An alternative for regression is the Mean Absolute Error (MAE). It calculates the average of the absolute differences between predictions and targets.

L_{MAE} = \frac{1}{N} \sum_{i=1}^{N} |y_i - \hat{y}_i|

MAE measures the average magnitude of the errors without considering their direction. Unlike MSE, it penalizes errors linearly. This makes MAE less sensitive to outliers compared to MSE, which might be beneficial in datasets with significant noise or anomalies. However, its gradient is constant, which can sometimes lead to issues with convergence when using gradient descent, especially when the loss is close to the minimum.

Loss Functions for Classification Problems

Classification problems involve predicting a discrete category or class label, such as identifying spam emails, classifying images, or predicting customer churn (yes/no).

Binary Cross-Entropy (Log Loss)

When dealing with binary classification (two possible output classes, typically labeled 0 and 1), the standard loss function is Binary Cross-Entropy, often called Log Loss. It's typically used when the output layer of the network has a single neuron with a Sigmoid activation function, which squashes the output to a probability between 0 and 1.

The formula for Binary Cross-Entropy for $N$ examples is:

L_{BCE} = - \frac{1}{N} \sum_{i=1}^{N} [y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i)]

Where:

$y_i$ is the true label (0 or 1) for the $i$ -th data point.
$\hat{y}_i$ is the predicted probability (output of the Sigmoid function, between 0 and 1) that the $i$ -th data point belongs to class 1.

Let's break this down:

If the true label $y_i$ is 1: The second term $(1 - y_i) \log(1 - \hat{y}_i)$ becomes zero. The loss becomes $- \log(\hat{y}_i)$ . If the network correctly predicts a probability $\hat{y}_i$ close to 1, $\log(\hat{y}_i)$ is close to 0, and the loss is small. If the network incorrectly predicts a probability $\hat{y}_i$ close to 0, $\log(\hat{y}_i)$ approaches negative infinity, and the loss becomes very large.
If the true label $y_i$ is 0: The first term $y_i \log(\hat{y}_i)$ becomes zero. The loss becomes $- \log(1 - \hat{y}_i)$ . If the network correctly predicts $\hat{y}_i$ close to 0, then $1 - \hat{y}_i$ is close to 1, $\log(1 - \hat{y}_i)$ is close to 0, and the loss is small. If the network incorrectly predicts $\hat{y}_i$ close to 1, then $1 - \hat{y}_i$ is close to 0, $\log(1 - \hat{y}_i)$ approaches negative infinity, and the loss becomes very large.

Binary Cross-Entropy heavily penalizes predictions that are confidently wrong.

Binary Cross-Entropy loss increases dramatically as the predicted probability moves away from the true label (0 or 1).

Categorical Cross-Entropy

For multi-class classification problems (more than two classes), we use Categorical Cross-Entropy. This requires the true labels $y$ to be in a one-hot encoded format (e.g., class 2 out of 3 classes would be [0, 0, 1]). The network's output layer typically uses a Softmax activation function, which produces a probability distribution across all $C$ classes, ensuring the predicted probabilities sum to 1.

The formula for Categorical Cross-Entropy for $N$ examples and $C$ classes is:

L_{CCE} = - \frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{C} y_{ij} \log(\hat{y}_{ij})

Where:

$N$ is the number of data points.
$C$ is the number of classes.
$y_{ij}$ is 1 if data point $i$ truly belongs to class $j$ , and 0 otherwise (from one-hot encoding).
$\hat{y}_{ij}$ is the network's predicted probability that data point $i$ belongs to class $j$ (output of the Softmax function).

Because $y_{ij}$ is 1 for only the single correct class and 0 for all others, the inner sum simplifies to just $- \log(\hat{y}_{ik})$ where $k$ is the index of the true class for data point $i$ . The loss function effectively measures how low the predicted probability is for the actual correct class. A higher predicted probability for the true class results in a lower loss.

Choosing the Right Loss Function

Selecting the appropriate loss function is fundamental to successful model training:

For regression tasks predicting continuous values, MSE is the standard choice, though MAE can be considered if outliers are a major concern.
For binary classification (two classes), use Binary Cross-Entropy, typically paired with a Sigmoid output activation.
For multi-class classification (more than two classes), use Categorical Cross-Entropy, typically paired with a Softmax output activation and one-hot encoded labels.

The loss function provides the error signal that drives learning. By calculating how far the network's predictions deviate from the true targets, it sets the stage for the optimization process. The next step is understanding how algorithms like gradient descent use this error signal to adjust the network's parameters.