Revisiting the Chain Rule for Single Variables

Machine learning models, especially neural networks, are often built by composing functions together. Imagine a simple process: an input $x$ goes into a function $g$, producing an output $u$. This output $u$ then becomes the input to another function $f$, yielding the final result $y$. Mathematically, we write this as $y = f(u)$ where $u = g(x)$, or more concisely as $y = h(x) = f(g(x))$.

Now, suppose we want to know how a small change in the initial input $x$ affects the final output $y$. This requires finding the derivative of the composite function $h(x)$ with respect to $x$, denoted as $\frac{dy}{dx}$ or $h'(x)$. We already know how to find $\frac{df}{du}$ (how $f$ changes with its direct input $u$) and $\frac{dg}{dx}$ (how $g$ changes with its input $x$). The chain rule provides the connection.

The chain rule for single-variable functions states that the derivative of a composite function $h(x) = f(g(x))$ is the product of the derivative of the outer function $f$ with respect to its argument (evaluated at the inner function $g(x)$) and the derivative of the inner function $g$ with respect to $x$.

Using Leibniz notation, which is often helpful for visualizing the "chain" effect, we write:

$$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$$

Alternatively, using Lagrange notation:

$$h'(x) = f'(g(x)) \cdot g'(x)$$

Let's break down $f'(g(x))$. It means:

1. Find the derivative of the outer function $f$ with respect to its input variable (let's call it $u$, so find $f'(u)$).
2. Substitute the inner function $g(x)$ back in place of $u$ in the expression for $f'(u)$.

Then, multiply this result by the derivative of the inner function, $g'(x)$.

Example 1: Polynomial Composition

Consider the function $h(x) = (x^2 + 5)^3$. This is a composition of functions. Let the inner function be $g(x) = u = x^2 + 5$. Let the outer function be $f(u) = y = u^3$.

First, find the derivatives of the individual functions:

• Derivative of the inner function: $\frac{du}{dx} = g'(x) = 2x$
• Derivative of the outer function: $\frac{dy}{du} = f'(u) = 3u^2$

Now, apply the chain rule: $\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$. Substitute $f'(u)$ and $g'(x)$:

$$\frac{dy}{dx} = (3u^2) \cdot (2x)$$

Finally, substitute the expression for $u$ back into the equation, as the derivative $f'(u)$ needs to be evaluated at the inner function's output ($u = g(x)$):

$$\frac{dy}{dx} = 3(x^2 + 5)^2 \cdot (2x) = 6x(x^2 + 5)^2$$

Example 2: Exponential Function

Let's look at $h(x) = e^{3x}$. We can see this as $y = f(u) = e^u$ and $u = g(x) = 3x$.

Find the individual derivatives:

• $\frac{du}{dx} = g'(x) = 3$
• $\frac{dy}{du} = f'(u) = e^u$ (The derivative of $e^u$ with respect to $u$ is $e^u$)

Apply the chain rule: $\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$

$$\frac{dy}{dx} = (e^u) \cdot (3)$$

Substitute $u = 3x$:

$$\frac{dy}{dx} = e^{3x} \cdot 3 = 3e^{3x}$$

This rule is fundamental because it allows us to break down the differentiation of complex, nested functions into manageable steps. We compute the rate of change layer by layer, multiplying these rates together to find the overall rate of change. This exact principle, extended to functions with multiple variables, forms the core mechanism of backpropagation in neural networks, allowing us to calculate how changes in weights deep inside the network affect the final output or error. We'll explore this multivariable extension next.