Updating Weights and Biases

To improve a neural network, the gradient of the loss function with respect to each weight and bias (represented as $\frac{\partial Loss}{\partial W}$ and $\frac{\partial Loss}{\partial b}$ for all parameters) is utilized. These gradients, typically determined through backpropagation, indicate the direction of the steepest increase in the loss. To minimize the loss, parameters must be adjusted in the opposite direction of their respective gradients.

This adjustment process is the core of gradient descent. For each weight ($W$) and bias ($b$) in the network, we update its value based on its current value, the calculated gradient, and a parameter called the learning rate ($\eta$).

The update rules are straightforward:

For a specific weight $W_{ij}$ (connecting neuron $i$ in one layer to neuron $j$ in the next): $W_{ij, new} = W_{ij, old} - \eta \frac{\partial Loss}{\partial W_{ij, old}}$

Similarly, for a specific bias $b_j$ (of neuron $j$): $b_{j, new} = b_{j, old} - \eta \frac{\partial Loss}{\partial b_{j, old}}$

Let's break down these equations:

• $W_{ij, old}$ and $b_{j, old}$ are the current values of the weight and bias before the update.
• $W_{ij, new}$ and $b_{j, new}$ are the updated values after applying the adjustment.
• $\frac{\partial Loss}{\partial W_{ij, old}}$ and $\frac{\partial Loss}{\partial b_{j, old}}$ are the gradients calculated during backpropagation. They represent how much the loss function changes for a small change in that specific weight or bias. A positive gradient means increasing the parameter increases the loss, while a negative gradient means increasing the parameter decreases the loss.
• $\eta$ (eta) is the learning rate. This is a small positive scalar value (e.g., 0.01, 0.001) that controls the size of the step we take in the direction opposite to the gradient. Choosing an appropriate learning rate is important for successful training, a topic we will discuss further in the next section.

The subtraction sign is fundamental here. If the gradient $\frac{\partial Loss}{\partial W_{ij, old}}$ is positive (meaning increasing $W_{ij}$ increases the loss), the update rule subtracts a positive value ($\eta \times \text{gradient}$), thereby decreasing $W_{ij}$. Conversely, if the gradient is negative (meaning increasing $W_{ij}$ decreases the loss), the update rule subtracts a negative value, which results in increasing $W_{ij}$. In both cases, the adjustment pushes the parameter value in a direction expected to lower the overall loss.

Vectorized Updates

In practice, performing these updates individually for every single parameter would be computationally inefficient. Deep learning frameworks leverage linear algebra libraries (like NumPy, TensorFlow, or PyTorch) to perform these updates using matrix and vector operations, applying the same logic simultaneously to all weights and biases within a layer or even the entire network.

If we represent all weights in a layer as a matrix $W$ and all biases as a vector $b$, and their corresponding gradients as $\nabla_W Loss$ and $\nabla_b Loss$, the update rules can be expressed concisely as:

$W_{new} = W_{old} - \eta \nabla_{W_{old}} Loss$ $b_{new} = b_{old} - \eta \nabla_{b_{old}} Loss$

Here, $\nabla_{W_{old}} Loss$ is a matrix of the same dimensions as $W_{old}$, where each element is the partial derivative of the loss with respect to the corresponding weight. Similarly, $\nabla_{b_{old}} Loss$ is a vector containing the partial derivatives with respect to each bias term. The subtraction and scalar multiplication ($\eta$) are performed element-wise.

This update step is typically performed once per batch of training data (in Mini-batch Gradient Descent) or after processing the entire dataset (in Batch Gradient Descent). This iterative process of calculating predictions (forward propagation), measuring error (loss function), computing gradients (backpropagation), and adjusting parameters (weight/bias updates) constitutes the core training loop for neural networks, gradually guiding the network towards a configuration that minimizes the prediction error on the training data.