Directional Derivatives

Partial derivatives measure a function's rate of change along the axes (like moving purely east or purely north on a map). The gradient vector, $\nabla f$, points in the direction of the steepest ascent (the direction to climb the hill fastest).

But what if you want to know how steep the terrain is in a different specific direction? Perhaps you want to move northeast, or along some arbitrary path represented by a vector. This is precisely what the directional derivative allows us to calculate. It quantifies the rate of change of a multivariable function $f$ at a particular point $\mathbf{a}$ when moving in a direction specified by a unit vector $\mathbf{u}$.

Defining the Directional Derivative

The directional derivative of a function $f$ at a point $\mathbf{a}$ in the direction of a unit vector $\mathbf{u}$ is denoted as $D_{\mathbf{u}}f(\mathbf{a})$. It's calculated using the dot product of the gradient at that point and the direction vector:

$$D_{\mathbf{u}}f(\mathbf{a}) = \nabla f(\mathbf{a}) \cdot \mathbf{u}$$

Let's break this down:

1. Gradient $\nabla f(\mathbf{a})$: As we learned, this vector contains all the partial derivatives of $f$ evaluated at $\mathbf{a}$. For a function $f(x, y)$, $\nabla f(\mathbf{a}) = \langle \frac{\partial f}{\partial x}(\mathbf{a}), \frac{\partial f}{\partial y}(\mathbf{a}) \rangle$. It encapsulates the function's rate of change information in all axis directions at point $\mathbf{a}$.
2. Unit Vector $\mathbf{u}$: This vector specifies the direction of interest. It's important that $\mathbf{u}$ is a unit vector, meaning its length or magnitude is 1 ($||\mathbf{u}|| = 1$). Why? Because we only want to capture the change due to the direction itself, not scaled by the length of the direction vector. If you have a direction specified by a vector $\mathbf{v}$ which is not a unit vector, you must first normalize it by dividing by its magnitude: $\mathbf{u} = \frac{\mathbf{v}}{||\mathbf{v}||}$.
3. Dot Product ($\cdot$): The dot product effectively measures how much one vector "goes in the direction of" another.

Geometric Intuition: Projection

Recall that the dot product between two vectors $\mathbf{v}_1$ and $\mathbf{v}_2$ can also be expressed as $\mathbf{v}_1 \cdot \mathbf{v}_2 = ||\mathbf{v}_1|| ||\mathbf{v}_2|| \cos \theta$, where $\theta$ is the angle between them.

Applying this to our directional derivative formula, and knowing that $||\mathbf{u}|| = 1$, we get:

$$D_{\mathbf{u}}f(\mathbf{a}) = \nabla f(\mathbf{a}) \cdot \mathbf{u} = ||\nabla f(\mathbf{a})|| \, ||\mathbf{u}|| \cos \theta = ||\nabla f(\mathbf{a})|| \cos \theta$$

Here, $\theta$ is the angle between the gradient vector $\nabla f(\mathbf{a})$ and the direction vector $\mathbf{u}$. This formula tells us something insightful: the directional derivative is the scalar projection of the gradient vector onto the direction vector $\mathbf{u}$. It's like asking, "How much of the gradient's magnitude points in the direction $\mathbf{u}$?"

The directional derivative $D_{\mathbf{u}}f$ is the scalar projection of the gradient vector $\nabla f$ onto the unit direction vector $\mathbf{u}$. It measures the component of the gradient acting in the direction $\mathbf{u}$.

This projection view helps understand the relationship between the gradient and the directional derivative:

• Maximum Change: When $\mathbf{u}$ points in the same direction as $\nabla f(\mathbf{a})$, the angle $\theta$ is 0, $\cos \theta = 1$, and $D_{\mathbf{u}}f(\mathbf{a}) = ||\nabla f(\mathbf{a})||$. The directional derivative is maximized and equals the magnitude of the gradient. This confirms the gradient points in the direction of steepest ascent.
• Minimum Change (Steepest Descent): When $\mathbf{u}$ points directly opposite to $\nabla f(\mathbf{a})$, the angle $\theta$ is $\pi$ (180 degrees), $\cos \theta = -1$, and $D_{\mathbf{u}}f(\mathbf{a}) = -||\nabla f(\mathbf{a})||$. This is the direction of steepest descent.
• Zero Change: When $\mathbf{u}$ is orthogonal (perpendicular) to $\nabla f(\mathbf{a})$, the angle $\theta$ is $\pi/2$ (90 degrees), $\cos \theta = 0$, and $D_{\mathbf{u}}f(\mathbf{a}) = 0$. Moving in this direction results in zero instantaneous change in the function's value. Geometrically, you are moving along a level curve or contour line on the function's surface.

Example Calculation

Let's consider the function $f(x, y) = x^2 + y^2$, which describes a parabolic bowl centered at the origin. We want to find the rate of change at the point $\mathbf{a} = (1, 1)$ in the direction of the vector $\mathbf{v} = \langle 1, 2 \rangle$.

1. Calculate the gradient: $\nabla f(x, y) = \langle \frac{\partial f}{\partial x}, \frac{\partial f}{\partial y} \rangle = \langle 2x, 2y \rangle$.
2. Evaluate the gradient at $\mathbf{a} = (1, 1)$: $\nabla f(1, 1) = \langle 2(1), 2(1) \rangle = \langle 2, 2 \rangle$. This vector points directly away from the origin, the direction of steepest ascent for this bowl shape.
3. Find the unit vector $\mathbf{u}$ for direction $\mathbf{v} = \langle 1, 2 \rangle$:
   • Magnitude: $||\mathbf{v}|| = \sqrt{1^2 + 2^2} = \sqrt{1 + 4} = \sqrt{5}$.
   • Normalize: $\mathbf{u} = \frac{\mathbf{v}}{||\mathbf{v}||} = \langle \frac{1}{\sqrt{5}}, \frac{2}{\sqrt{5}} \rangle$.
4. Calculate the directional derivative using the dot product: $D_{\mathbf{u}}f(1, 1) = \nabla f(1, 1) \cdot \mathbf{u} = \langle 2, 2 \rangle \cdot \langle \frac{1}{\sqrt{5}}, \frac{2}{\sqrt{5}} \rangle$ $D_{\mathbf{u}}f(1, 1) = (2)(\frac{1}{\sqrt{5}}) + (2)(\frac{2}{\sqrt{5}}) = \frac{2}{\sqrt{5}} + \frac{4}{\sqrt{5}} = \frac{6}{\sqrt{5}}$.

So, at the point $(1, 1)$, if we move in the direction $\langle 1, 2 \rangle$, the function $f(x, y)$ increases at a rate of $\frac{6}{\sqrt{5}} \approx 2.68$ units per unit distance moved. Notice this is less than the magnitude of the gradient, $||\nabla f(1, 1)|| = ||\langle 2, 2 \rangle|| = \sqrt{2^2 + 2^2} = \sqrt{8} \approx 2.83$, which is the rate of change in the steepest direction $\langle 2, 2 \rangle$.

Relevance in Machine Learning

In machine learning optimization, especially with gradient descent, we are primarily interested in the direction of steepest descent, which is $-\nabla f$. However, understanding directional derivatives provides valuable context about the parameter space. It helps us reason about why moving in the direction of the negative gradient is the most efficient step (locally) to minimize the loss function. While not typically calculated explicitly during standard gradient descent, the concept underpins our understanding of how the function behaves in the high-dimensional parameter space we navigate during model training. It reinforces the central role of the gradient in guiding optimization.