Gradient Descent Algorithm

Loss functions, such as Mean Squared Error ($MSE$) or Cross-Entropy, quantify the discrepancy between a neural network's predictions and the actual values. The primary objective of training is to minimize this error. To achieve this, the network's weights and biases must be adjusted to reduce the loss. This adjustment process is handled by optimization algorithms, and the most fundamental among them is Gradient Descent.

Imagine you are standing on a foggy mountain range, and your goal is to reach the lowest valley (the minimum loss). You can only see the ground right beneath your feet. How would you proceed? A sensible strategy would be to feel the slope of the ground where you are and take a step in the steepest downhill direction. You repeat this process, taking step after step downhill, hoping to eventually reach the bottom.

Gradient Descent works in a very similar way. The "mountain range" is the loss function's surface, defined by the network's parameters (weights $w$ and biases $b$, collectively denoted as $\theta$). The "altitude" at any point is the value of the loss function for a given set of parameters $\theta$. Our goal is to find the parameters $\theta$ that correspond to the lowest point on this surface, the minimum loss $J(\theta)$.

Calculus gives us a tool to find the "steepest direction": the gradient. The gradient of the loss function with respect to the parameters, denoted as $\nabla J(\theta)$, is a vector that points in the direction of the steepest ascent on the loss surface. Since we want to go downhill to minimize the loss, we take steps in the direction opposite to the gradient (the negative gradient, $-\nabla J(\theta)$).

The core idea of Gradient Descent is to iteratively update the parameters by taking small steps in the negative gradient direction. The update rule for a single parameter (or the entire set of parameters $\theta$) looks like this:

$$\theta_{new} = \theta_{old} - \alpha \nabla J(\theta_{old})$$

Let's break this down:

• $\theta_{old}$: The current value of the parameters (weights and biases) before the update.
• $\theta_{new}$: The updated value of the parameters after taking one step.
• $\nabla J(\theta_{old})$: The gradient of the loss function calculated using the current parameters $\theta_{old}$. This tells us the direction of steepest increase in loss.
• $\alpha$: This is the learning rate. It's a small positive number (e.g., 0.01, 0.001) that controls how big a step we take in the negative gradient direction. If the step is too large, we might overshoot the minimum. If it's too small, the process might take too long. We'll discuss the learning rate in more detail in the next section.

So, the process involves:

1. Initializing the parameters ($\theta$) randomly or using a specific initialization strategy.
2. Calculating the loss $J(\theta)$ based on the current parameters and the training data.
3. Calculating the gradient of the loss function with respect to each parameter ($\nabla J(\theta)$). For neural networks, this calculation typically involves an algorithm called backpropagation (which we'll cover in Chapter 4).
4. Updating each parameter using the update rule: $\theta_{new} = \theta_{old} - \alpha \nabla J(\theta_{old})$.
5. Repeating steps 2-4 for a set number of iterations (epochs) or until the loss stops decreasing significantly (convergence).

Let's visualize this on a simple 1D loss curve, like $J(w) = w^2$. The minimum is clearly at $w=0$. The gradient is $\frac{dJ}{dw} = 2w$.

A simple quadratic loss function $J(w) = w^2$. Starting at $w=4$, Gradient Descent iteratively takes steps proportional to the negative gradient ($-2w$) multiplied by the learning rate ($\alpha=0.2$) towards the minimum at $w=0$.

In practice, for neural networks, the loss surface is much more complex and high-dimensional (one dimension for each weight and bias). Calculating the gradient $\nabla J(\theta)$ involves computing the partial derivative of the loss with respect to every single weight and bias in the network. While simple, the actual computation requires the chain rule and the backpropagation algorithm.

A potential issue with the basic version described here (often called Batch Gradient Descent) is that calculating $\nabla J(\theta)$ requires evaluating the loss and its gradient over the entire training dataset in each step. For large datasets, this can be extremely slow and computationally expensive. This motivates variations like Stochastic Gradient Descent (SGD) and Mini-Batch Gradient Descent, which we will discuss shortly.

For now, the significant takeaway is that Gradient Descent provides the mechanism for learning. By repeatedly calculating the direction of steepest descent (negative gradient) on the loss surface and taking steps in that direction, it iteratively adjusts the network's parameters to minimize the prediction error.