Common Differentiation Rules

The limit definition of a derivative, $\lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$, provides a fundamental way to calculate derivatives. However, applying this definition directly can be tedious for complex functions. Imagine applying this definition repeatedly within a machine learning optimization loop. It would be computationally inefficient. Fortunately, calculus provides a set of rules that allow us to find derivatives much more easily by breaking down complex functions into simpler parts. These rules are the building blocks for differentiating the kinds of functions we encounter when defining model costs or activation functions in machine learning.

Let's explore the most common and useful differentiation rules. Assume $f(x)$ and $g(x)$ are differentiable functions, and $c$ is a constant.

The Constant Rule

The simplest rule involves functions that always output the same value, like $f(x) = 5$. Since the function's value doesn't change, its rate of change (the derivative) is always zero.

• Rule: If $f(x) = c$, where $c$ is a constant, then $f'(x) = 0$.
• In Leibniz notation: $\frac{d}{dx}(c) = 0$
• Example: If $f(x) = 10$, then $f'(x) = 0$.

The Power Rule

This is one of the most frequently used rules, especially for polynomial functions common in modeling. It applies to functions of the form $x$ raised to a power.

• Rule: If $f(x) = x^n$, where $n$ is any real number, then $f'(x) = nx^{n-1}$.
• In Leibniz notation: $\frac{d}{dx}(x^n) = nx^{n-1}$
• Explanation: To differentiate $x^n$, you bring the exponent $n$ down as a multiplier and then decrease the original exponent by 1.
• Examples:
  • If $f(x) = x^3$, then $f'(x) = 3x^{3-1} = 3x^2$.
  • If $g(x) = x$ (which is $x^1$), then $g'(x) = 1x^{1-1} = 1x^0 = 1$. This makes sense, the slope of the line $y=x$ is 1.
  • If $h(x) = \sqrt{x} = x^{1/2}$, then $h'(x) = \frac{1}{2}x^{1/2 - 1} = \frac{1}{2}x^{-1/2} = \frac{1}{2\sqrt{x}}$.
  • If $k(x) = \frac{1}{x^2} = x^{-2}$, then $k'(x) = -2x^{-2-1} = -2x^{-3} = -\frac{2}{x^3}$.

The Constant Multiple Rule

What happens if the function is multiplied by a constant, like $f(x) = 5x^3$? This rule states that you can essentially pull the constant out, differentiate the rest of the function, and then multiply the constant back in.

• Rule: If $h(x) = c \cdot f(x)$, then $h'(x) = c \cdot f'(x)$.
• In Leibniz notation: $\frac{d}{dx}(c \cdot f(x)) = c \cdot \frac{d}{dx}(f(x))$
• Example: If $f(x) = 5x^3$. We know $\frac{d}{dx}(x^3) = 3x^2$. Using the rule: $f'(x) = 5 \cdot \frac{d}{dx}(x^3) = 5 \cdot (3x^2) = 15x^2$

The Sum/Difference Rule

Machine learning cost functions are often sums of terms (like the sum of squared errors). This rule allows us to differentiate such functions term by term.

• Rule: If $h(x) = f(x) + g(x)$, then $h'(x) = f'(x) + g'(x)$. Similarly, if $h(x) = f(x) - g(x)$, then $h'(x) = f'(x) - g'(x)$.
• In Leibniz notation: $\frac{d}{dx}(f(x) \pm g(x)) = \frac{d}{dx}(f(x)) \pm \frac{d}{dx}(g(x))$
• Explanation: The derivative of a sum (or difference) is the sum (or difference) of the derivatives.
• Example: Let $h(x) = x^4 + 6x^2 - 7$. We can differentiate term by term using the Power Rule, Constant Multiple Rule, and Constant Rule: $h'(x) = \frac{d}{dx}(x^4) + \frac{d}{dx}(6x^2) - \frac{d}{dx}(7)$ $h'(x) = (4x^3) + (6 \cdot 2x) - (0)$ $h'(x) = 4x^3 + 12x$

The Product Rule

This rule is needed when differentiating a function that is the product of two other functions, like $h(x) = f(x)g(x)$. It's important not to simply multiply the individual derivatives.

• Rule: If $h(x) = f(x)g(x)$, then $h'(x) = f'(x)g(x) + f(x)g'(x)$.
• In Leibniz notation: $\frac{d}{dx}(f(x)g(x)) = \frac{df}{dx}g(x) + f(x)\frac{dg}{dx}$
• Explanation: The derivative of a product is the derivative of the first function times the second function plus the first function times the derivative of the second function.
• Example: Let $h(x) = (x^2+1)(x^3-x)$.
  • Let $f(x) = x^2+1$, so $f'(x) = 2x$.
  • Let $g(x) = x^3-x$, so $g'(x) = 3x^2-1$.
  • Applying the Product Rule: $h'(x) = f'(x)g(x) + f(x)g'(x)$ $h'(x) = (2x)(x^3-x) + (x^2+1)(3x^2-1)$ $h'(x) = (2x^4 - 2x^2) + (3x^4 - x^2 + 3x^2 - 1)$ $h'(x) = 2x^4 - 2x^2 + 3x^4 + 2x^2 - 1$ $h'(x) = 5x^4 - 1$
  • (Self-check: Expand $h(x)$ first: $h(x) = x^5 - x^3 + x^3 - x = x^5 - x$. Then $h'(x) = 5x^4 - 1$. The rule works!)

The Quotient Rule

This rule handles functions that are formed by dividing one function by another, $h(x) = \frac{f(x)}{g(x)}$. Like the product rule, it has a specific structure.

• Rule: If $h(x) = \frac{f(x)}{g(x)}$, where $g(x) \neq 0$, then $h'(x) = \frac{f'(x)g(x) - f(x)g'(x)}{[g(x)]^2}$.
• In Leibniz notation: $\frac{d}{dx}\left(\frac{f(x)}{g(x)}\right) = \frac{\frac{df}{dx}g(x) - f(x)\frac{dg}{dx}}{[g(x)]^2}$
• Explanation: A common mnemonic is "low d-high minus high d-low, square the bottom and away we go," where "low" is $g(x)$, "high" is $f(x)$, and "d-" means "the derivative of".
• Example: Let $h(x) = \frac{x^2}{x-1}$.
  • Let $f(x) = x^2$ (high), so $f'(x) = 2x$.
  • Let $g(x) = x-1$ (low), so $g'(x) = 1$.
  • Applying the Quotient Rule: $h'(x) = \frac{f'(x)g(x) - f(x)g'(x)}{[g(x)]^2}$ $h'(x) = \frac{(2x)(x-1) - (x^2)(1)}{(x-1)^2}$ $h'(x) = \frac{2x^2 - 2x - x^2}{(x-1)^2}$ $h'(x) = \frac{x^2 - 2x}{(x-1)^2}$

These rules, particularly the Power, Constant Multiple, and Sum/Difference rules, are fundamental for differentiating the polynomial-like terms often found in cost functions (like Mean Squared Error). The Product and Quotient rules become essential when dealing with more complex model structures or activation functions. Mastering these rules allows us to analytically compute the derivatives needed for optimization without resorting to the limit definition every time. Later, we'll see how the Chain Rule combines with these to handle even more complex, nested functions, which are characteristic of neural networks.