Revisiting Gradient Descent

To train a deep learning model, our objective is to find the set of parameters (weights and biases) that minimizes a specific loss function for our given data. Optimization algorithms are the tools we use to perform this search efficiently. The most fundamental algorithm, forming the foundation upon which many others are built, is Gradient Descent.

You'll likely recall Gradient Descent from introductory machine learning concepts. Its core idea is straightforward: iteratively adjust the model's parameters in the direction that maximally reduces the loss function. How do we know which direction reduces the loss the most? We use the gradient.

The Gradient: Our Guide Downhill

The loss function, let's call it $J(\theta)$, measures how poorly our model performs with its current parameters $\theta$. This function lives in a high-dimensional space (one dimension for each parameter). We want to find the point in this space corresponding to the lowest loss.

The gradient of the loss function with respect to the parameters, denoted as $\nabla J(\theta)$, is a vector pointing in the direction of the steepest ascent of the loss function at point $\theta$. Since we want to minimize the loss, we should move in the direction opposite to the gradient.

Imagine you are standing on a foggy hill and want to get to the valley floor (minimum elevation, or minimum loss). You can't see the whole area, but you can feel the slope under your feet. The gradient tells you the direction of the steepest uphill path. To go down, you take a step in the exact opposite direction.

The Update Rule

Gradient Descent formalizes this intuition with a simple update rule. In each step, we update the parameters $\theta$ as follows:

$$\theta_{\text{new}} = \theta_{\text{old}} - \alpha \nabla J(\theta_{\text{old}})$$

Let's break this down:

• $\theta_{\text{old}}$: The current values of our model parameters.
• $\nabla J(\theta_{\text{old}})$: The gradient of the loss function calculated using the current parameters $\theta_{\text{old}}$. This vector tells us the direction of steepest increase in loss.
• $\alpha$: The learning rate. This is a small positive scalar hyperparameter that controls the size of the step we take in the opposite direction of the gradient. Choosing a good learning rate is important: too small, and training will be very slow; too large, and we might overshoot the minimum or even diverge.
• $\theta_{\text{new}}$: The updated parameter values after taking one step.

We repeat this update process iteratively, calculating the gradient and updating the parameters, hoping to converge to a set of parameters $\theta$ that results in a low value for the loss function $J(\theta)$.

A simplified 2D loss surface (contours represent equal loss values). The red line shows the path taken by Gradient Descent, starting from an initial point and taking steps opposite to the gradient towards the minimum (darkest blue area).

Batch Gradient Descent

The "standard" version of Gradient Descent, often referred to as Batch Gradient Descent (BGD), has a specific characteristic: to calculate the gradient $\nabla J(\theta)$ in each update step, it processes the entire training dataset. It computes the average loss and the average gradient over all training examples before making a single parameter update.

This gives a very accurate estimate of the true gradient for the entire dataset, leading to a smooth convergence path towards a minimum. However, as mentioned in the chapter introduction, computing the gradient over the entire dataset can be computationally prohibitive for the massive datasets commonly used in deep learning. Imagine calculating predictions and gradients for millions of images just to make one tiny adjustment to the model's weights!

This computational burden is a primary motivation for the variations of Gradient Descent we will explore next, such as Stochastic Gradient Descent (SGD) and Mini-batch Gradient Descent. These methods offer ways to get faster updates, albeit with some trade-offs, making the optimization process feasible for large-scale deep learning. Understanding the mechanics of Batch Gradient Descent provides the necessary foundation for appreciating why these variations were developed and how they work.