The Gradient Vector

Partial derivatives provide the means to understand how functions with multiple inputs, like $f(x, y)$, change. For instance, $\frac{\partial f}{\partial x}$ describes how $f$ changes as $x$ varies, keeping $y$ fixed. Similarly, $\frac{\partial f}{\partial y}$ indicates how $f$ changes as $y$ varies, while $x$ is held constant.

But what if we want a single object that captures the rate of change with respect to all input variables simultaneously? That's where the gradient vector comes in.

The gradient of a function $f$ is simply a vector where each component is a partial derivative of $f$. If our function has two inputs, $x$ and $y$, its gradient is a two-dimensional vector. If it has $n$ inputs, $x_1, x_2, \dots, x_n$, its gradient is an $n$-dimensional vector.

We usually denote the gradient using the nabla symbol, $\nabla$. For a function $f(x, y)$, the gradient is written as $\nabla f$ or $\nabla f(x, y)$ and is defined as:

$$\nabla f(x, y) = \begin{bmatrix} \frac{\partial f}{\partial x} \\ \frac{\partial f}{\partial y} \end{bmatrix}$$

Sometimes you might also see it written horizontally using angle brackets: $\nabla f(x, y) = \langle \frac{\partial f}{\partial x}, \frac{\partial f}{\partial y} \rangle$.

For a function with $n$ variables, $f(x_1, x_2, \dots, x_n)$, the gradient is:

$$\nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \frac{\partial f}{\partial x_2} \\ \vdots \\ \frac{\partial f}{\partial x_n} \end{bmatrix}$$

Essentially, the gradient packages up all the first-order partial derivatives into one convenient vector.

Example: Calculating a Gradient

Let's take the function $f(x, y) = x^2 + 5xy$.

1. Find the partial derivatives:

   • Treat $y$ as a constant to find $\frac{\partial f}{\partial x}$: $$\frac{\partial f}{\partial x} = \frac{\partial}{\partial x}(x^2) + \frac{\partial}{\partial x}(5xy) = 2x + 5y$$
   • Treat $x$ as a constant to find $\frac{\partial f}{\partial y}$: $$\frac{\partial f}{\partial y} = \frac{\partial}{\partial y}(x^2) + \frac{\partial}{\partial y}(5xy) = 0 + 5x = 5x$$

2. Assemble the gradient vector:

   $$\nabla f(x, y) = \begin{bmatrix} \frac{\partial f}{\partial x} \\ \frac{\partial f}{\partial y} \end{bmatrix} = \begin{bmatrix} 2x + 5y \\ 5x \end{bmatrix}$$

This gradient, $\nabla f(x, y)$, gives us a vector that depends on the point $(x, y)$ we are considering. For instance, at the point $(1, 2)$:

$$\nabla f(1, 2) = \begin{bmatrix} 2(1) + 5(2) \\ 5(1) \end{bmatrix} = \begin{bmatrix} 2 + 10 \\ 5 \end{bmatrix} = \begin{bmatrix} 12 \\ 5 \end{bmatrix}$$

At the point $(-1, 0)$:

$$\nabla f(-1, 0) = \begin{bmatrix} 2(-1) + 5(0) \\ 5(-1) \end{bmatrix} = \begin{bmatrix} -2 \\ -5 \end{bmatrix}$$

So, the gradient is itself a function that takes a point $(x, y)$ as input and outputs a vector. This vector holds important information about how the function $f$ behaves around that specific point, which we'll examine more closely in the next section. In machine learning, the gradient of a cost function tells us how to adjust model parameters (like the inputs $x_1, x_2, \dots, x_n$) to change the cost.