Backpropagation: Calculating Gradients Efficiently

Okay, we know we need to calculate the gradient of the loss function with respect to each weight and bias in the network. This gradient tells us the direction of steepest ascent of the loss function; we want to move in the opposite direction (gradient descent) to minimize the error. The central challenge is computing these gradients efficiently, especially for networks with many layers and parameters. Trying to calculate the impact of each individual parameter on the final loss by slightly changing it (numerical differentiation) is computationally infeasible for practical networks.

This is where the backpropagation algorithm comes in. It's a remarkably efficient method for computing all the necessary gradients by leveraging the chain rule from calculus. The name "backpropagation" comes from the fact that it computes the gradients starting from the final output layer and works its way backward through the network to the input layer, propagating the error gradient as it goes.

The Power of the Chain Rule

At its heart, a neural network is a series of nested functions. The output of one layer becomes the input to the next. To find how a change in a parameter deep inside the network affects the final loss, we need to apply the chain rule repeatedly.

Consider a very simple network calculation:

1. Input $x$
2. Hidden layer pre-activation: $z^{(1)} = w^{(1)}x + b^{(1)}$
3. Hidden layer activation: $a^{(1)} = \sigma(z^{(1)})$ (where $\sigma$ is an activation function)
4. Output layer pre-activation: $z^{(2)} = w^{(2)}a^{(1)} + b^{(2)}$
5. Output layer activation (prediction): $a^{(2)} = \sigma(z^{(2)})$
6. Loss: $L = \text{Loss}(a^{(2)}, y)$ (where $y$ is the true label)

If we want to find the gradient of the loss $L$ with respect to a weight in the first layer, say $w^{(1)}$, we need to trace its influence forward to the loss: $w^{(1)}$ affects $z^{(1)}$, which affects $a^{(1)}$, which affects $z^{(2)}$, which affects $a^{(2)}$, which finally affects $L$. The chain rule allows us to break this down:

$$\frac{\partial L}{\partial w^{(1)}} = \frac{\partial L}{\partial a^{(2)}} \times \frac{\partial a^{(2)}}{\partial z^{(2)}} \times \frac{\partial z^{(2)}}{\partial a^{(1)}} \times \frac{\partial a^{(1)}}{\partial z^{(1)}} \times \frac{\partial z^{(1)}}{\partial w^{(1)}}$$

Backpropagation organizes this calculation efficiently. It calculates terms like $\frac{\partial L}{\partial a^{(2)}}$ and $\frac{\partial L}{\partial z^{(2)}}$ first, and then reuses them to compute gradients for earlier layers.

The Backward Pass Explained

The algorithm works in two main phases:

1. Forward Pass: Input data is fed through the network layer by layer, calculating pre-activations ($z^{(l)}$) and activations ($a^{(l)}$) at each step, ultimately producing the final output prediction $a^{(L)}$. During this pass, we store the intermediate values ($z^{(l)}$ and $a^{(l)}$ for all layers $l$) as they will be needed for the backward pass. The final loss $L$ is calculated using the prediction $a^{(L)}$ and the true target $y$.

2. Backward Pass: This pass computes the gradients.

   • Output Layer: Calculate the gradient of the loss with respect to the output layer's activation, $\frac{\partial L}{\partial a^{(L)}}$. Then, using the chain rule, calculate the gradient with respect to the output layer's pre-activation $z^{(L)}$: $\delta^{(L)} = \frac{\partial L}{\partial z^{(L)}} = \frac{\partial L}{\partial a^{(L)}} \frac{\partial a^{(L)}}{\partial z^{(L)}} = \frac{\partial L}{\partial a^{(L)}} \sigma'(z^{(L)})$ Here, $\sigma'(z^{(L)})$ is the derivative of the activation function used in the output layer, evaluated at $z^{(L)}$. This term $\delta^{(L)}$ represents the "error signal" at the output layer's pre-activation. Now, calculate the gradients for the output layer's weights $W^{(L)}$ and biases $b^{(L)}$: $\frac{\partial L}{\partial W^{(L)}} = \frac{\partial L}{\partial z^{(L)}} \frac{\partial z^{(L)}}{\partial W^{(L)}} = \delta^{(L)} (a^{(L-1)})^T$ $\frac{\partial L}{\partial b^{(L)}} = \frac{\partial L}{\partial z^{(L)}} \frac{\partial z^{(L)}}{\partial b^{(L)}} = \delta^{(L)}$ (Note: The exact forms depend on whether we are using vector/matrix notation. $a^{(L-1)}$ represents the activations from the previous layer feeding into the output layer. The transpose ensures dimensions match for matrix multiplication).

   • Hidden Layers (working backward from $l = L-1$ down to $l=1$): For each hidden layer $l$, calculate its error signal $\delta^{(l)}$ based on the error signal $\delta^{(l+1)}$ from the next layer (the one closer to the output): $\delta^{(l)} = \frac{\partial L}{\partial z^{(l)}} = \left( (W^{(l+1)})^T \delta^{(l+1)} \right) \odot \sigma'(z^{(l)})$ Here, $(W^{(l+1)})^T$ are the weights connecting layer $l$ to layer $l+1$. This step effectively propagates the error gradient backward through the network weights. The term $\sigma'(z^{(l)})$ is the derivative of the activation function of the current hidden layer $l$. The symbol $\odot$ denotes element-wise multiplication (Hadamard product). Once $\delta^{(l)}$ is known, calculate the gradients for layer $l$'s weights $W^{(l)}$ and biases $b^{(l)}$ similar to the output layer: $\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (a^{(l-1)})^T$ $\frac{\partial L}{\partial b^{(l)}} = \delta^{(l)}$ Where $a^{(l-1)}$ are the activations from the layer feeding into layer $l$ (for the first hidden layer, this would be the input data $x$).

This process continues backward until gradients for all parameters have been computed.

Visualizing the Flow

We can think of the computation as flowing forward to get the loss, and then backward to get the gradients.

A simplified view of forward (blue arrows) and backward (red arrows) passes. The forward pass computes activations and loss. The backward pass computes gradients, starting from the loss and propagating error signals (like $\delta^{(l)} = \frac{\partial L}{\partial z^{(l)}}$) backward, reusing values calculated during the forward pass ($a^{(l)}$, $z^{(l)}$) and network weights ($W^{(l)}$). Gradients for parameters ($W, b$) are derived from these error signals.

Why is it Efficient?

The efficiency of backpropagation comes from two main aspects:

1. Reusing Calculations: Intermediate values ($z^{(l)}$, $a^{(l)}$) computed during the forward pass are stored and reused during the backward pass. More importantly, the error signal ($\delta^{(l+1)}$) computed for layer $l+1$ is directly used to compute the error signal ($\delta^{(l)}$) for layer $l$. This avoids redundant calculations.
2. Dynamic Programming: It essentially uses dynamic programming. By calculating gradients layer by layer from back to front, it builds up the solution for the entire network's gradients without needing to recompute the influence paths for every single parameter independently.

Compared to numerically estimating gradients, which requires at least one extra forward pass for each parameter, backpropagation computes all gradients in roughly the same amount of computational time as a single forward pass. For a network with millions of parameters, this difference is enormous, making the training of deep networks feasible.

With the gradients computed via backpropagation, we now have the direction needed to update the weights and biases using gradient descent, which we will detail in the following sections.