In the previous chapter, we focused on functions with a single input variable, like $f(x)$ . We learned how derivatives tell us about the rate of change of these functions. However, many situations in machine learning involve functions that depend on multiple inputs simultaneously. Think about predicting a house price. The price doesn't just depend on the square footage; it might also depend on the number of bedrooms, the age of the house, and its location. Similarly, the cost function we aim to minimize during model training usually depends on all the model's parameters (weights and biases), which can number in the thousands or even millions for complex models.

To handle these scenarios, we need to extend our calculus toolkit to functions of multiple variables.

What are Functions of Multiple Variables?

A function of multiple variables takes more than one input value and produces an output value (often a single scalar value in our context). If a function $f$ depends on $n$ variables, say $x_1, x_2, \dots, x_n$ , we write it as:

$f(x_1, x_2, \dots, x_n)$

Alternatively, we can group the input variables into a vector $\mathbf{x} = [x_1, x_2, \dots, x_n]^T$ . Then, the function can be written more compactly as:

$f(\mathbf{x})$

The output is typically a single real number, especially when dealing with cost functions in machine learning, where the output represents a measure of error or 'cost'.

Example: Consider a simple function of two variables, $x$ and $y$ :

$f(x, y) = x^2 + 2y^2$

Here, the input consists of a pair of values $(x, y)$ , and the output is a single number calculated using the formula. For instance, $f(1, 2) = 1^2 + 2(2^2) = 1 + 8 = 9$ .

Visualizing Multivariable Functions

Visualizing functions becomes more challenging as the number of input variables increases.

One Variable ( $y = f(x)$ ): We can easily visualize this as a curve on a 2D plane (x-axis and y-axis).
Two Variables ( $z = f(x, y)$ ): We can visualize this as a surface in 3D space, where the height $z$ above the $xy$ -plane represents the function's value at point $(x, y)$ .

Let's visualize the function $z = x^2 + y^2$ . This equation describes a paraboloid, opening upwards, with its minimum value at $(0, 0)$ .

A 3D surface plot representing the function $z = x^2 + y^2$ . The height ( $z$ ) corresponds to the function's value for each pair of $(x, y)$ coordinates. The minimum value (0) occurs at $(x, y) = (0, 0)$ .

Three or More Variables ( $w = f(x, y, z, \dots)$ ): Direct visualization in 3D space is no longer possible. For functions of three variables, we might sometimes use level sets (surfaces in 3D where the function value is constant). For functions with more than three variables, we rely heavily on mathematical analysis rather than direct geometric intuition. We often analyze the function by looking at how it behaves along specific directions or by examining its cross-sections.

Relevance in Machine Learning

Functions of multiple variables are fundamental in machine learning:

Cost Functions: As mentioned, the cost function $J$ measures how well our model performs based on its parameters (weights $\mathbf{w}$ and bias $b$ ). For a model with $n$ weights, the cost function is $J(w_1, w_2, \dots, w_n, b)$ . Training the model involves finding the values of $w_1, \dots, w_n, b$ that minimize this multivariable function. For example, the mean squared error cost for linear regression with two features ( $x_1, x_2$ ) and parameters $w_1, w_2, b$ is: $J(w_1, w_2, b) = \frac{1}{2m} \sum_{i=1}^{m} ( (w_1 x_1^{(i)} + w_2 x_2^{(i)} + b) - y^{(i)} )^2$ Here, $m$ is the number of training examples, $(x_1^{(i)}, x_2^{(i)})$ are the features of the $i$ -th example, and $y^{(i)}$ is its true label. The function $J$ depends on the three parameters $w_1, w_2,$ and $b$ .
Model Predictions: The prediction function $h(\mathbf{x})$ itself is often a multivariable function. For linear regression, $h_{\mathbf{w}, b}(\mathbf{x}) = w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b$ . The prediction depends on the input features $x_1, \dots, x_n$ .

Understanding how to analyze these multivariable functions, particularly how they change as we adjust their inputs (the parameters), is essential for optimization. This naturally leads us to the concept of partial derivatives, which measure the rate of change with respect to one variable while holding others constant. We will explore this next.