The Gradient Descent Algorithm Steps

The Gradient Descent algorithm is an iterative optimization process that formalizes the steps for descending a cost function to find a minimum.

The core idea is to repeatedly adjust the model parameters, typically denoted by the vector $\theta$, in a direction that decreases the cost function $J(\theta)$. Since the gradient $\nabla J(\theta)$ points in the direction of the steepest ascent, we move in the opposite direction.

The Update Rule

Gradient descent updates each parameter $\theta_j$ (where $j$ indexes the different parameters in our model, like weights and biases) using the following rule in each iteration:

$$\theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j} J(\theta)$$

Let's dissect this important update step:

1. $\theta_j$: This is the current value of the $j$-th parameter we want to optimize. Our goal is to find the values for all $\theta_j$ that minimize the overall cost $J(\theta)$.
2. $:=$: This denotes assignment. The value of $\theta_j$ is updated with the result calculated on the right-hand side.
3. $\alpha$: This is the learning rate, a hyperparameter that controls the size of the step we take during each iteration. A smaller $\alpha$ means smaller steps, potentially leading to slower convergence but more stability. A larger $\alpha$ can speed up convergence but risks overshooting the minimum or even diverging. We'll examine the learning rate in more detail in the next section.
4. $\frac{\partial}{\partial \theta_j} J(\theta)$: This is the partial derivative of the cost function $J(\theta)$ with respect to the parameter $\theta_j$. It represents the slope of the cost function along the axis of $\theta_j$ at the current point defined by all parameters $\theta$. This term tells us how much the cost function $J$ changes if we slightly change $\theta_j$, holding all other parameters constant. It's one component of the gradient vector $\nabla J(\theta)$. The sign of this derivative indicates whether increasing $\theta_j$ increases or decreases the cost. By subtracting $\alpha$ times this derivative, we are nudged in the direction that decreases the cost.

The Iterative Process

Gradient descent is an iterative algorithm. It doesn't typically find the optimal parameters in a single step. Instead, it follows these general stages:

1. Initialization: Start with initial guesses for the parameters $\theta$. These could be zeros, small random values, or values from a pre-trained model.
2. Iteration: Repeat the following until a stopping criterion (convergence) is met:
   • Compute the Gradient: Calculate the partial derivative term $\frac{\partial}{\partial \theta_j} J(\theta)$ for all parameters $j = 0, 1, ..., n$, based on the current parameter values $\theta$. The exact data used for this calculation depends on the specific variant of gradient descent (Batch, Stochastic, or Mini-batch), which we'll discuss later in this chapter.
   • Update Parameters: Apply the update rule simultaneously for all parameters: $$\theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j} J(\theta) \quad \text{for } j = 0, 1, ..., n$$
3. Convergence: Stop when the parameters are no longer changing significantly, the cost function $J(\theta)$ decrease becomes very small, the gradient magnitude is close to zero, or a maximum number of iterations is reached.

Simultaneous Updates: A Point of Attention

It's important that the updates for all parameters $\theta_j$ within a single iteration happen simultaneously. This means you should first compute the partial derivatives for all $j$ using the current parameter values from the start of the iteration. Only after all these gradient components are calculated should you update the values of $\theta_0, \theta_1, ..., \theta_n$.

Consider a model with two parameters, $\theta_0$ and $\theta_1$. The correct update procedure for one iteration looks like this:

# Calculate gradient components using current theta0, theta1
gradient_component_0 = compute_partial_derivative(J, theta0, theta1, wrt=theta0)
gradient_component_1 = compute_partial_derivative(J, theta1, theta0, wrt=theta1)

# Calculate the new values using the computed gradients
temp0 = theta0 - alpha * gradient_component_0
temp1 = theta1 - alpha * gradient_component_1

# Update the parameters simultaneously
theta0 = temp0
theta1 = temp1

Updating $\theta_0$ first and then immediately using this new $\theta_0$ to calculate the gradient component for $\theta_1$ in the same iteration is incorrect for standard gradient descent. Store the updates temporarily or use vector operations to ensure simultaneous application.

Here is a concise pseudocode representation of the core algorithm:

Initialize parameters theta (e.g., randomly or with zeros)
Repeat until convergence {
  Compute gradient_components[j] = partial_derivative(J, theta, wrt=theta[j]) for all j
  For j = 0 to n {
    theta[j] = theta[j] - alpha * gradient_components[j]
  }
}
Return theta

These steps form the foundation of how gradient descent navigates the cost surface. The effectiveness and behavior of this process, however, heavily depend on factors like the learning rate $\alpha$ and how we compute the gradient, which we will explore next.