Practice: Calculating Partial Derivatives and Gradients

Partial derivatives describe the rate of change of a multi-variable function with respect to one variable, while holding the others constant. The gradient vector combines these partial derivatives, pointing in the direction of the function's steepest ascent. Practice in calculating partial derivatives and gradients for various functions is provided.

Remember, the process relies heavily on the derivative rules you learned earlier (like the power rule and sum rule), with one extra step: treating certain variables as constants during differentiation.

Example 1: A Simple Polynomial Function

Let's start with a function of two variables, $x$ and $y$:

$$f(x, y) = 2x^3 + 4xy^2 - y^5 + 7$$

Our goal is to find the partial derivatives, $\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$, and then form the gradient vector, $\nabla f(x, y)$.

Calculating $\frac{\partial f}{\partial x}$ (Partial Derivative with respect to $x$)

To find $\frac{\partial f}{\partial x}$, we treat $y$ as if it were a constant number (like 5, or -2, or $\pi$).

1. Term 1 ($2x^3$): The derivative of $2x^3$ with respect to $x$ is $2 \cdot (3x^{3-1}) = 6x^2$. (Standard power rule).
2. Term 2 ($4xy^2$): Here, we treat $4$ and $y^2$ as constants multiplied by $x$. The derivative of (constant $\times x$) with respect to $x$ is just the constant. So, the derivative of $4xy^2$ with respect to $x$ is $4y^2$.
3. Term 3 ($-y^5$): Since we treat $y$ as a constant, $y^5$ is also a constant. The derivative of any constant with respect to $x$ is 0.
4. Term 4 ($7$): This is a constant, so its derivative with respect to $x$ is 0.

Putting it together:

$$\frac{\partial f}{\partial x} = 6x^2 + 4y^2 + 0 + 0 = 6x^2 + 4y^2$$

Calculating $\frac{\partial f}{\partial y}$ (Partial Derivative with respect to $y$)

Now, we switch perspectives. To find $\frac{\partial f}{\partial y}$, we treat $x$ as if it were a constant.

1. Term 1 ($2x^3$): Since $x$ is treated as constant, $2x^3$ is also constant. Its derivative with respect to $y$ is 0.
2. Term 2 ($4xy^2$): Treat $4x$ as the constant coefficient of $y^2$. The derivative of (constant $\times y^2$) with respect to $y$ is (constant $\times 2y$). So, the derivative is $4x \cdot (2y) = 8xy$.
3. Term 3 ($-y^5$): The derivative of $-y^5$ with respect to $y$ is $-5y^{5-1} = -5y^4$. (Standard power rule).
4. Term 4 ($7$): This is a constant, so its derivative with respect to $y$ is 0.

Putting it together:

$$\frac{\partial f}{\partial y} = 0 + 8xy - 5y^4 + 0 = 8xy - 5y^4$$

Forming the Gradient Vector $\nabla f(x, y)$

The gradient vector is simply a vector containing the partial derivatives. By convention, we list them in the order of the variables (x, then y).

$$\nabla f(x, y) = \begin{bmatrix} \frac{\partial f}{\partial x} \\ \frac{\partial f}{\partial y} \end{bmatrix} = \begin{bmatrix} 6x^2 + 4y^2 \\ 8xy - 5y^4 \end{bmatrix}$$

Evaluating the Gradient at a Point

The gradient itself is a function; it gives us the direction of steepest ascent at any point $(x, y)$. Let's find the gradient at the specific point $(x, y) = (1, 2)$. We substitute $x=1$ and $y=2$ into our expressions for the partial derivatives:

• $\frac{\partial f}{\partial x}$ at $(1, 2)$: $6(1)^2 + 4(2)^2 = 6(1) + 4(4) = 6 + 16 = 22$
• $\frac{\partial f}{\partial y}$ at $(1, 2)$: $8(1)(2) - 5(2)^4 = 16 - 5(16) = 16 - 80 = -64$

So, the gradient vector at the point $(1, 2)$ is:

$$\nabla f(1, 2) = \begin{bmatrix} 22 \\ -64 \end{bmatrix}$$

This vector tells us that starting from the point $(1, 2)$ on the surface defined by $f(x, y)$, the direction of steepest increase is primarily along the positive x-axis (value 22) and strongly along the negative y-axis (value -64). In optimization (like gradient descent), we would typically move in the opposite direction, $-\nabla f(1, 2)$, to decrease the function's value.

Example 2: Function with Interaction Term

Let's try another one. Consider a function $g(w_1, w_2)$ which might represent a simple cost function depending on two weights, $w_1$ and $w_2$.

$$g(w_1, w_2) = 3w_1^2 - 5w_1w_2 + 2w_2^2$$

Calculating $\frac{\partial g}{\partial w_1}$

Treat $w_2$ as a constant.

1. Term 1 ($3w_1^2$): Derivative w.r.t. $w_1$ is $3(2w_1) = 6w_1$.
2. Term 2 ($-5w_1w_2$): Treat $-5w_2$ as the constant coefficient of $w_1$. Derivative w.r.t. $w_1$ is $-5w_2$.
3. Term 3 ($2w_2^2$): Treat $w_2$ as constant, so $2w_2^2$ is constant. Derivative w.r.t. $w_1$ is 0.

Result:

$$\frac{\partial g}{\partial w_1} = 6w_1 - 5w_2$$

Calculating $\frac{\partial g}{\partial w_2}$

Treat $w_1$ as a constant.

1. Term 1 ($3w_1^2$): Treat $w_1$ as constant, so $3w_1^2$ is constant. Derivative w.r.t. $w_2$ is 0.
2. Term 2 ($-5w_1w_2$): Treat $-5w_1$ as the constant coefficient of $w_2$. Derivative w.r.t. $w_2$ is $-5w_1$.
3. Term 3 ($2w_2^2$): Derivative w.r.t. $w_2$ is $2(2w_2) = 4w_2$.

Result:

$$\frac{\partial g}{\partial w_2} = -5w_1 + 4w_2$$

Forming the Gradient Vector $\nabla g(w_1, w_2)$

$$\nabla g(w_1, w_2) = \begin{bmatrix} \frac{\partial g}{\partial w_1} \\ \frac{\partial g}{\partial w_2} \end{bmatrix} = \begin{bmatrix} 6w_1 - 5w_2 \\ -5w_1 + 4w_2 \end{bmatrix}$$

Evaluating the Gradient at a Point

Let's evaluate the gradient at $(w_1, w_2) = (2, -1)$.

• $\frac{\partial g}{\partial w_1}$ at $(2, -1)$: $6(2) - 5(-1) = 12 + 5 = 17$
• $\frac{\partial g}{\partial w_2}$ at $(2, -1)$: $-5(2) + 4(-1) = -10 - 4 = -14$

So, the gradient vector at the point $(2, -1)$ is:

$$\nabla g(2, -1) = \begin{bmatrix} 17 \\ -14 \end{bmatrix}$$

Why Practice This?

Calculating partial derivatives and gradients is a fundamental mechanical skill needed for understanding and implementing optimization algorithms in machine learning. When we train models, we often have a cost function that depends on many parameters (weights and biases). Gradient descent uses the gradient of this cost function to iteratively update the parameters in the direction that minimizes the cost. You'll see this process in action in the next chapter.

Spend some time working through these examples and perhaps try creating and differentiating a few simple functions of your own. The more comfortable you are with these calculations, the clearer the optimization process will become.