The Backpropagation Algorithm Explained

Training a neural network involves adjusting its weights and biases to minimize a loss function, typically using gradient descent. The primary requirement for gradient descent is the gradient of the loss function with respect to every parameter (weight $w_{ij}^{(l)}$ and bias $b_i^{(l)}$) in the network. Calculating these gradients naively for a deep network with potentially millions of parameters would be computationally prohibitive.

This is where the backpropagation algorithm comes in. It's not a new optimization algorithm itself, but rather a remarkably efficient method for computing the necessary gradients. Backpropagation elegantly applies the chain rule of calculus, working backward from the final loss calculation through the network layers to determine how each parameter contributes to that loss.

Think of it like assigning "blame" or "credit" for the final error. If the network's output is far from the target, backpropagation helps determine which connections (weights) and thresholds (biases) were most responsible for the discrepancy, and in which direction they should be adjusted.

The Two Passes of Backpropagation

The algorithm operates in two main passes:

1. Forward Pass: This is the standard process of feeding an input $x$ through the network, layer by layer, calculating the weighted sums ($z^{(l)}$) and activations ($a^{(l)}$) at each layer until we reach the final output activation $a^{(L)}$ and compute the loss $L$. During this pass, it's essential to store the intermediate values, specifically the weighted sums $z^{(l)}$ and activations $a^{(l)}$ for each layer $l$, as they are needed for the backward pass.

   • $z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$
   • $a^{(l)} = g(z^{(l)})$ (where $g$ is the activation function)

2. Backward Pass: This is the core of backpropagation. It starts at the output layer and propagates the gradient of the loss backward through the network.

   • Output Layer (Layer L): We first compute the gradient of the loss with respect to the output activations, $\frac{\partial L}{\partial a^{(L)}}$. The exact form depends on the loss function used. Then, using the chain rule and the stored $z^{(L)}$ value, we compute the gradient of the loss with respect to the pre-activation value $z^{(L)}$ for the output layer. This term is often denoted as the "error" $\delta^{(L)}$ for the output layer:

     $$\delta^{(L)} = \frac{\partial L}{\partial z^{(L)}} = \frac{\partial L}{\partial a^{(L)}} \odot g'(z^{(L)})$$

     Here, $g'$ is the derivative of the activation function used in the output layer, and $\odot$ represents element-wise multiplication (Hadamard product). This is because both $\frac{\partial L}{\partial a^{(L)}}$ and $g'(z^{(L)})$ are vectors (or tensors) of the same dimension as the output layer.

   • Gradients for Output Layer Parameters: Once we have $\delta^{(L)}$, we can compute the gradients for the weights $W^{(L)}$ and biases $b^{(L)}$ connecting to the output layer:

     $$\frac{\partial L}{\partial W^{(L)}} = \delta^{(L)} (a^{(L-1)})^T$$ $$\frac{\partial L}{\partial b^{(L)}} = \delta^{(L)}$$

     (Note: For $\frac{\partial L}{\partial W^{(L)}}$, this is an outer product if $\delta^{(L)}$ and $a^{(L-1)}$ are vectors. Summation over the batch dimension is implicit if using mini-batches).

   • Propagating the Error Backward: The important step is to calculate how the error $\delta^{(L)}$ propagates back to the previous layer's activations, $a^{(L-1)}$. We need $\frac{\partial L}{\partial a^{(L-1)}}$ to continue the process. This is found by considering how $a^{(L-1)}$ influences $z^{(L)}$ (via $W^{(L)}$) and then how $z^{(L)}$ influences $L$ (via $\delta^{(L)}$):

     $$\frac{\partial L}{\partial a^{(L-1)}} = (W^{(L)})^T \delta^{(L)}$$

   • Hidden Layers (Layer l): Now we generalize the process for any hidden layer $l$ (moving backward from $L-1$ down to 1). We assume we have already computed $\frac{\partial L}{\partial a^{(l)}}$ (which is the error propagated from layer $l+1$). We compute the error $\delta^{(l)}$ for the current layer:

     $$\delta^{(l)} = \frac{\partial L}{\partial z^{(l)}} = \frac{\partial L}{\partial a^{(l)}} \odot g'(z^{(l)}) = \left( (W^{(l+1)})^T \delta^{(l+1)} \right) \odot g'(z^{(l)})$$

     This equation shows how the error from the next layer ($\delta^{(l+1)}$), weighted by the connections ($W^{(l+1)}$), is combined with the local gradient of the activation function ($g'(z^{(l)})$) to form the error for the current layer.

   • Gradients for Hidden Layer Parameters: Using $\delta^{(l)}$, we compute the gradients for the parameters of layer $l$:

     $$\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (a^{(l-1)})^T$$ $$\frac{\partial L}{\partial b^{(l)}} = \delta^{(l)}$$

   • Continue Backward: We repeat this process, calculating $\delta^{(l)}$, $\frac{\partial L}{\partial W^{(l)}}$, and $\frac{\partial L}{\partial b^{(l)}}$ for each layer $l$ from $L-1$ down to $1$. Each step uses the error $\delta^{(l+1)}$ computed in the previous step (the step for the layer $l+1$).

Visualizing the Flow

The diagram below illustrates the backward flow of gradient information. The forward pass computes values from left to right. The backward pass (dashed red lines) computes gradients starting from the loss $L$ and moves right to left, calculating the required partial derivatives ($\partial L / \partial \dots$) at each stage.

Flow of backpropagation. Solid lines represent the forward pass computations. Dashed lines represent the backward flow of gradient calculations, starting from the Loss (L) and moving leftward through the layers. Each dashed edge label indicates the gradient being computed at that step.

Efficiency Gain

The efficiency of backpropagation comes from reusing calculations. Notice how the error $\delta^{(l)}$ calculated for layer $l$ is directly used to compute both the gradients $\frac{\partial L}{\partial W^{(l)}}$, $\frac{\partial L}{\partial b^{(l)}}$ and the error term $\delta^{(l-1)}$ needed for the previous layer (via $(W^{(l)})^T \delta^{(l)}$). This avoids redundant computations that would occur if we tried to calculate each $\frac{\partial L}{\partial W_{ij}}$ and $\frac{\partial L}{\partial b_i}$ independently from scratch using the chain rule all the way back to the input.

In essence, backpropagation is a practical application of reverse-mode automatic differentiation applied to neural networks. It provides a structured way to compute the gradient of a scalar objective function (the loss) with respect to all its input parameters (the weights and biases) in a time complexity roughly proportional to the cost of the forward pass itself.

Once the backward pass is complete, we have the gradients $\frac{\partial L}{\partial W^{(l)}}$ and $\frac{\partial L}{\partial b^{(l)}}$ for all layers $l$. These gradients are then used by an optimization algorithm (like SGD, Adam, etc.) to update the network's parameters:

$W^{(l)} \leftarrow W^{(l)} - \eta \frac{\partial L}{\partial W^{(l)}}$ $b^{(l)} \leftarrow b^{(l)} - \eta \frac{\partial L}{\partial b^{(l)}}$

where $\eta$ is the learning rate. This completes one cycle of forward pass, loss calculation, backward pass (backpropagation), and parameter update, forming the basis of how neural networks learn.