Calculating Gradients for Weights and Biases

Okay, we've established that backpropagation is essentially the chain rule applied systematically backward through the network layers. The ultimate goal of this process is to figure out how much a small change in each weight and bias affects the overall loss function $L$. This information, captured by the gradients $\frac{\partial L}{\partial W}$ and $\frac{\partial L}{\partial b}$, is exactly what gradient descent needs to update the parameters and improve the model.

Let's zoom in on a specific layer, say layer $l$, within the network. During the forward pass, this layer took the activation $a^{[l-1]}$ from the previous layer (or the input features if $l=1$) and computed its own output activation $a^{[l]}$ through two steps:

1. Pre-activation: $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$
2. Activation: $a^{[l]} = g^{[l]}(z^{[l]})$

Here, $W^{[l]}$ and $b^{[l]}$ are the weight matrix and bias vector for layer $l$, and $g^{[l]}$ is its activation function.

The backpropagation algorithm works layer by layer, starting from the final layer and moving toward the input. When we reach layer $l$ during the backward pass, we assume we have already calculated $\frac{\partial L}{\partial a^{[l]}}$, the gradient of the loss with respect to the output activation of this layer. Our task now is to use this information to compute:

1. $\frac{\partial L}{\partial W^{[l]}}$: Gradient of the loss with respect to the weights of layer $l$.
2. $\frac{\partial L}{\partial b^{[l]}}$: Gradient of the loss with respect to the biases of layer $l$.
3. $\frac{\partial L}{\partial a^{[l-1]}}$: Gradient of the loss with respect to the input activation of layer $l$. This is needed to continue the backward pass to layer $l-1$.

Let's see how the chain rule helps us compute these quantities.

Calculating the Gradient with Respect to Pre-activation ($z^{[l]}$)

First, we need the gradient of the loss with respect to the pre-activation $z^{[l]}$. Since $L$ depends on $a^{[l]}$, and $a^{[l]}$ depends directly on $z^{[l]}$ via the activation function $g^{[l]}$, we apply the chain rule:

$$\frac{\partial L}{\partial z^{[l]}} = \frac{\partial L}{\partial a^{[l]}} \frac{\partial a^{[l]}}{\partial z^{[l]}}$$

Recall that $a^{[l]} = g^{[l]}(z^{[l]})$. Therefore, the second term is simply the derivative of the activation function, $g'^{[l]}(z^{[l]})$.

$$\frac{\partial L}{\partial z^{[l]}} = \frac{\partial L}{\partial a^{[l]}} g'^{[l]}(z^{[l]})$$

This gradient, $\frac{\partial L}{\partial z^{[l]}}$, represents how the loss changes with respect to the linear combination calculated just before the activation function in layer $l$. It's a significant intermediate value often denoted as $\delta^{[l]}$ or, in code, dZ[l]. It essentially carries the error signal backward through the activation function's derivative.

Calculating the Gradients for Weights ($W^{[l]}$) and Biases ($b^{[l]}$)

Now that we have $\frac{\partial L}{\partial z^{[l]}}$, we can find the gradients for the parameters $W^{[l]}$ and $b^{[l]}$. Remember that $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$.

For Weights ($W^{[l]}$): The loss $L$ depends on $z^{[l]}$, which in turn depends on $W^{[l]}$. Applying the chain rule:

$$\frac{\partial L}{\partial W^{[l]}} = \frac{\partial L}{\partial z^{[l]}} \frac{\partial z^{[l]}}{\partial W^{[l]}}$$

We need the partial derivative of $z^{[l]}$ with respect to $W^{[l]}$. Looking at $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$, the derivative $\frac{\partial z^{[l]}}{\partial W^{[l]}}$ turns out to be $a^{[l-1]}$ (specifically, its transpose in matrix notation, due to the rules of matrix calculus). Intuitively, how much $z^{[l]}$ changes when $W^{[l]}$ changes depends on the input activation $a^{[l-1]}$ it was multiplied by.

So, the gradient for the weights is:

$$\frac{\partial L}{\partial W^{[l]}} = \frac{\partial L}{\partial z^{[l]}} (a^{[l-1]})^T$$

In implementations using vectorized operations (handling multiple examples in a batch, denoted by $m$), this is often calculated as:

$$dW^{[l]} = \frac{1}{m} dZ^{[l]} (A^{[l-1]})^T$$

where $dZ^{[l]}$ represents $\frac{\partial L}{\partial Z^{[l]}}$ (the matrix form of $\frac{\partial L}{\partial z^{[l]}}$), $A^{[l-1]}$ contains the activations from the previous layer for all examples in the batch, and $dW^{[l]}$ is the resulting gradient matrix for the weights. The $\frac{1}{m}$ factor averages the gradient over the batch.

For Biases ($b^{[l]}$): Similarly, $L$ depends on $z^{[l]}$, which depends on $b^{[l]}$.

$$\frac{\partial L}{\partial b^{[l]}} = \frac{\partial L}{\partial z^{[l]}} \frac{\partial z^{[l]}}{\partial b^{[l]}}$$

The derivative $\frac{\partial z^{[l]}}{\partial b^{[l]}}$ is simply 1, because $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ changes by exactly the same amount as $b^{[l]}$ changes (holding other variables constant).

$$\frac{\partial L}{\partial b^{[l]}} = \frac{\partial L}{\partial z^{[l]}} \cdot 1 = \frac{\partial L}{\partial z^{[l]}}$$

So, the gradient for the bias is just the gradient with respect to the pre-activation $z^{[l]}$. When working with batches, the gradient $dZ^{[l]}$ will typically have dimensions corresponding to (number of neurons in layer $l$, number of examples $m$). The bias $b^{[l]}$ is usually a vector applied to all examples. Therefore, the gradient $db^{[l]}$ is computed by summing (or averaging) $dZ^{[l]}$ across the batch dimension:

$$db^{[l]} = \frac{1}{m} \sum_{i=1}^{m} (dZ^{[l]})^{(i)}$$

where $(dZ^{[l]})^{(i)}$ represents the $i$-th column (example) of the $dZ^{[l]}$ matrix.

Calculating the Gradient for the Previous Layer's Activation ($a^{[l-1]}$)

Finally, to continue the backward pass, we need to compute $\frac{\partial L}{\partial a^{[l-1]}}$. This tells layer $l-1$ how its output activations affected the overall loss. Again, we use the chain rule, noting that $L$ depends on $z^{[l]}$, which depends on $a^{[l-1]}$.

$$\frac{\partial L}{\partial a^{[l-1]}} = \frac{\partial L}{\partial z^{[l]}} \frac{\partial z^{[l]}}{\partial a^{[l-1]}}$$

From $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$, the derivative $\frac{\partial z^{[l]}}{\partial a^{[l-1]}}$ is the weight matrix $W^{[l]}$ (specifically, its transpose according to matrix calculus rules). Intuitively, the influence of $a^{[l-1]}$ on $z^{[l]}$ is determined by the weights $W^{[l]}$ it was multiplied by.

$$\frac{\partial L}{\partial a^{[l-1]}} = (W^{[l]})^T \frac{\partial L}{\partial z^{[l]}}$$

In vectorized form:

$$dA^{[l-1]} = (W^{[l]})^T dZ^{[l]}$$

This $dA^{[l-1]}$ becomes the input gradient $\frac{\partial L}{\partial a^{[l-1]}}$ for the next step of backpropagation when we process layer $l-1$.

Summary of Layer $l$ Gradient Calculations

During the backward pass for layer $l$, assuming we have $dA^{[l]} = \frac{\partial L}{\partial A^{[l]}}$, we compute:

1. $dZ^{[l]} = dA^{[l]} * g'^{[l]}(Z^{[l]})$ (element-wise multiplication)
2. $dW^{[l]} = \frac{1}{m} dZ^{[l]} (A^{[l-1]})^T$
3. $db^{[l]} = \frac{1}{m} \text{np.sum}(dZ^{[l]}, \text{axis=1, keepdims=True})$ (using NumPy-like notation for summing across examples)
4. $dA^{[l-1]} = (W^{[l]})^T dZ^{[l]}$

This computed $dA^{[l-1]}$ is then passed backward to layer $l-1$, and the process repeats until we reach the input layer. This systematic application of the chain rule allows us to efficiently find the gradients for all weights and biases in the network. The diagram below illustrates this flow for a single layer.

Gradient computation flow for layer $l$, showing vectorized operations. The backward pass uses the incoming gradient $dA^{[l]}$ to compute $dZ^{[l]}$, which then allows calculation of the parameter gradients $dW^{[l]}$ and $db^{[l]}$ (yellow) and the gradient $dA^{[l-1]}$ (red) needed for the previous layer.