ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Calculus Fundamentals for Machine Learning
Chapter 1: Why Calculus for Machine Learning?
The Role of Math in Machine Learning
Introducing Functions: Inputs and Outputs
Visualizing Functions: Graphs
What is a Limit?
Intuition Behind Limits
Connecting Functions and Limits to ML
Quiz for Chapter 1
Chapter 2: Derivatives: Measuring Change
Rate of Change: Average vs Instantaneous
The Derivative: Slope of a Tangent Line
Derivative Notation (Leibniz and Lagrange)
Calculating Derivatives: The Power Rule
Calculating Derivatives: Constants and Sums
Introduction to Higher-Order Derivatives
Practice: Calculating Simple Derivatives
Quiz for Chapter 2
Chapter 3: Optimization with Derivatives
Finding Maximum and Minimum Points
Optimization: Why Minimize or Maximize?
Cost Functions in Machine Learning
Goal: Minimizing the Cost Function
Introduction to Gradient Descent
How Derivatives Guide Gradient Descent
Visualizing Gradient Descent
Quiz for Chapter 3
Chapter 4: Handling Multiple Inputs: Partial Derivatives
Functions of Multiple Variables
Partial Derivatives: The Concept
Calculating Partial Derivatives
Partial Derivative Notation
The Gradient Vector
Geometric Meaning of the Gradient
Practice: Calculating Partial Derivatives and Gradients
Quiz for Chapter 4
Chapter 5: Calculus in Action: Simple Optimization
Recap: Optimization Goal and Gradient Descent
Example: Simple Linear Regression Model
Defining a Cost Function for Linear Regression
Calculating Gradients for the Cost Function
Performing a Gradient Descent Step
The Learning Rate Parameter
Putting It All Together: The Optimization Process
Hands-on Practical: Manual Gradient Calculation
Quiz for Chapter 5

Calculating Gradients for the Cost Function

Calculating gradients for the cost function is a primary step in optimization. A simple linear model, y=mx+by = mx + by=mx+b, represents data. To measure how well this line fits the data, a cost function, typically the Mean Squared Error (MSE), quantifies the error. The objective is to find the values of mmm (slope) and bbb (y-intercept) that minimize this cost.

Gradient descent needs directions. It needs to know: "If I change mmm slightly, how does the cost change?" and "If I change bbb slightly, how does the cost change?" This is precisely what partial derivatives tell us. We need to calculate the gradient of the cost function, ∇C\nabla C∇C, which consists of the partial derivatives with respect to each parameter: mmm and bbb.

The Cost Function: Mean Squared Error (MSE)

Let's assume we have NNN data points, (x1,y1),(x2,y2),...,(xN,yN)(x_1, y_1), (x_2, y_2), ..., (x_N, y_N)(x1​,y1​),(x2​,y2​),...,(xN​,yN​). For any given mmm and bbb, our model predicts ypred,i=mxi+by_{pred, i} = mx_i + bypred,i​=mxi​+b for each input xix_ixi​. The actual value is yiy_iyi​.

The Mean Squared Error cost function C(m,b)C(m, b)C(m,b) is defined as the average of the squared differences between the predicted and actual values:

C(m,b)=1N∑i=1N(ypred,i−yactual,i)2C(m, b) = \frac{1}{N} \sum_{i=1}^{N} (y_{pred, i} - y_{actual, i})^2C(m,b)=N1​i=1∑N​(ypred,i​−yactual,i​)2

Substituting our model's prediction:

C(m,b)=1N∑i=1N((mxi+b)−yi)2C(m, b) = \frac{1}{N} \sum_{i=1}^{N} ( (mx_i + b) - y_i )^2C(m,b)=N1​i=1∑N​((mxi​+b)−yi​)2

This function C(m,b)C(m,b)C(m,b) takes mmm and bbb as inputs and outputs a single number representing the average error. Our task now is to find ∂C∂m\frac{\partial C}{\partial m}∂m∂C​ and ∂C∂b\frac{\partial C}{\partial b}∂b∂C​.

Calculating the Partial Derivative with Respect to mmm (∂C∂m\frac{\partial C}{\partial m}∂m∂C​)

To find ∂C∂m\frac{\partial C}{\partial m}∂m∂C​, we differentiate the cost function C(m,b)C(m, b)C(m,b) with respect to mmm, treating bbb (and all xix_ixi​, yiy_iyi​, and NNN) as constants.

Let's look at the expression term by term:

  1. Constant Factor: The 1N\frac{1}{N}N1​ is a constant multiplier, so it stays put.
  2. Sum Rule: The derivative of a sum is the sum of the derivatives. We can move the derivative inside the summation: ∂C∂m=1N∑i=1N∂∂m(mxi+b−yi)2\frac{\partial C}{\partial m} = \frac{1}{N} \sum_{i=1}^{N} \frac{\partial}{\partial m} (mx_i + b - y_i)^2∂m∂C​=N1​i=1∑N​∂m∂​(mxi​+b−yi​)2
  3. Chain Rule: We need to differentiate the term (mxi+b−yi)2(mx_i + b - y_i)^2(mxi​+b−yi​)2 with respect to mmm. Let u=mxi+b−yiu = mx_i + b - y_iu=mxi​+b−yi​. We are differentiating u2u^2u2 with respect to mmm. The chain rule states ddm(u2)=2u⋅∂u∂m\frac{d}{dm}(u^2) = 2u \cdot \frac{\partial u}{\partial m}dmd​(u2)=2u⋅∂m∂u​.
    • First part: 2u=2(mxi+b−yi)2u = 2(mx_i + b - y_i)2u=2(mxi​+b−yi​).
    • Second part: We need ∂u∂m=∂∂m(mxi+b−yi)\frac{\partial u}{\partial m} = \frac{\partial}{\partial m} (mx_i + b - y_i)∂m∂u​=∂m∂​(mxi​+b−yi​). Since bbb and yiy_iyi​ are treated as constants, their derivatives are zero. The derivative of mximx_imxi​ with respect to mmm is just xix_ixi​ (because xix_ixi​ is treated as a constant coefficient of mmm). So, ∂u∂m=xi\frac{\partial u}{\partial m} = x_i∂m∂u​=xi​.
  4. Putting it Together: Substituting back into the chain rule formula: ∂∂m(mxi+b−yi)2=2(mxi+b−yi)⋅xi\frac{\partial}{\partial m} (mx_i + b - y_i)^2 = 2(mx_i + b - y_i) \cdot x_i∂m∂​(mxi​+b−yi​)2=2(mxi​+b−yi​)⋅xi​.
  5. Final Result for ∂C∂m\frac{\partial C}{\partial m}∂m∂C​: Now substitute this back into the summation: ∂C∂m=1N∑i=1N2(mxi+b−yi)xi\frac{\partial C}{\partial m} = \frac{1}{N} \sum_{i=1}^{N} 2(mx_i + b - y_i) x_i∂m∂C​=N1​i=1∑N​2(mxi​+b−yi​)xi​ We can pull the constant 2 out of the sum: ∂C∂m=2N∑i=1N(mxi+b−yi)xi\frac{\partial C}{\partial m} = \frac{2}{N} \sum_{i=1}^{N} (mx_i + b - y_i) x_i∂m∂C​=N2​i=1∑N​(mxi​+b−yi​)xi​

This expression tells us how the cost changes as we slightly change the slope mmm. Notice it depends on the error term (mxi+b−yi)(mx_i + b - y_i)(mxi​+b−yi​) and the input value xix_ixi​ for each data point.

Calculating the Partial Derivative with Respect to bbb (∂C∂b\frac{\partial C}{\partial b}∂b∂C​)

Now we repeat the process, but this time we differentiate C(m,b)C(m, b)C(m,b) with respect to bbb, treating mmm as a constant.

  1. Constant Factor and Sum Rule: Same as before, we get: ∂C∂b=1N∑i=1N∂∂b(mxi+b−yi)2\frac{\partial C}{\partial b} = \frac{1}{N} \sum_{i=1}^{N} \frac{\partial}{\partial b} (mx_i + b - y_i)^2∂b∂C​=N1​i=1∑N​∂b∂​(mxi​+b−yi​)2
  2. Chain Rule: Again, let u=mxi+b−yiu = mx_i + b - y_iu=mxi​+b−yi​. We need ddb(u2)=2u⋅∂u∂b\frac{d}{db}(u^2) = 2u \cdot \frac{\partial u}{\partial b}dbd​(u2)=2u⋅∂b∂u​.
    • First part: 2u=2(mxi+b−yi)2u = 2(mx_i + b - y_i)2u=2(mxi​+b−yi​).
    • Second part: We need ∂u∂b=∂∂b(mxi+b−yi)\frac{\partial u}{\partial b} = \frac{\partial}{\partial b} (mx_i + b - y_i)∂b∂u​=∂b∂​(mxi​+b−yi​). This time, mmm, xix_ixi​, and yiy_iyi​ are treated as constants. The derivative of mximx_imxi​ with respect to bbb is 0. The derivative of bbb with respect to bbb is 1. The derivative of yiy_iyi​ with respect to bbb is 0. So, ∂u∂b=1\frac{\partial u}{\partial b} = 1∂b∂u​=1.
  3. Putting it Together: ∂∂b(mxi+b−yi)2=2(mxi+b−yi)⋅1\frac{\partial}{\partial b} (mx_i + b - y_i)^2 = 2(mx_i + b - y_i) \cdot 1∂b∂​(mxi​+b−yi​)2=2(mxi​+b−yi​)⋅1.
  4. Final Result for ∂C∂b\frac{\partial C}{\partial b}∂b∂C​: Substitute back into the summation: ∂C∂b=1N∑i=1N2(mxi+b−yi)(1)\frac{\partial C}{\partial b} = \frac{1}{N} \sum_{i=1}^{N} 2(mx_i + b - y_i) (1)∂b∂C​=N1​i=1∑N​2(mxi​+b−yi​)(1) Pulling the constant 2 out: ∂C∂b=2N∑i=1N(mxi+b−yi)\frac{\partial C}{\partial b} = \frac{2}{N} \sum_{i=1}^{N} (mx_i + b - y_i)∂b∂C​=N2​i=1∑N​(mxi​+b−yi​)

This expression tells us how the cost changes as we slightly change the y-intercept bbb. It depends only on the error term (mxi+b−yi)(mx_i + b - y_i)(mxi​+b−yi​) for each data point.

The Gradient Vector ∇C\nabla C∇C

We've successfully calculated the partial derivatives! We can now assemble them into the gradient vector for our cost function C(m,b)C(m, b)C(m,b):

∇C(m,b)=[∂C∂m,∂C∂b]\nabla C(m, b) = \left[ \frac{\partial C}{\partial m}, \frac{\partial C}{\partial b} \right]∇C(m,b)=[∂m∂C​,∂b∂C​]

Substituting the expressions we found:

∇C(m,b)=[2N∑i=1N(mxi+b−yi)xi,2N∑i=1N(mxi+b−yi)]\nabla C(m, b) = \left[ \frac{2}{N} \sum_{i=1}^{N} (mx_i + b - y_i) x_i, \quad \frac{2}{N} \sum_{i=1}^{N} (mx_i + b - y_i) \right]∇C(m,b)=[N2​i=1∑N​(mxi​+b−yi​)xi​,N2​i=1∑N​(mxi​+b−yi​)]

This vector, ∇C\nabla C∇C, is fundamental. For any given values of mmm and bbb:

  • The first component (∂C∂m\frac{\partial C}{\partial m}∂m∂C​) tells us the rate of change of the cost function if we move purely in the mmm direction. Its sign tells us if increasing mmm increases or decreases the cost.
  • The second component (∂C∂b\frac{\partial C}{\partial b}∂b∂C​) tells us the rate of change of the cost function if we move purely in the bbb direction. Its sign tells us if increasing bbb increases or decreases the cost.

Crucially, the gradient vector ∇C\nabla C∇C points in the direction of the steepest increase of the cost function C(m,b)C(m,b)C(m,b) at the current point (m,b)(m,b)(m,b). Since we want to minimize the cost, gradient descent will involve taking steps in the direction opposite to the gradient.

With these calculated gradients, we now have the precise information needed to iteratively update mmm and bbb to find the line that best fits our data. The next step is to see how these gradients are used within the gradient descent algorithm itself.

  • Mathematics for Machine Learning, Marc Peter Deisenroth, A. Aldo Faisal, Cheng Soon Ong, 2020 (Cambridge University Press) - Provides a solid mathematical foundation for machine learning, including coverage of calculus, partial derivatives, and optimization techniques relevant to cost functions.
  • Calculus: Early Transcendentals, James Stewart, 2015 (Cengage Learning) - A widely recognized textbook for multivariable calculus, offering rigorous explanations of partial derivatives, the chain rule, and the concept of gradients.
  • Lecture Notes: Linear Regression (CS229), Andrew Ng, Tengyu Ma, 2023 Stanford University CS229 Lecture Notes (Stanford University) - These lecture notes from a renowned machine learning course detail the derivation of cost functions and their gradients for linear regression, a direct application of the section's content.
  • Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - Chapter 4 covers numerical computation, including gradient-based optimization and the mathematical foundations for calculating gradients of objective functions.

© 2026 ApX Machine Learning