ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Introduction to Machine Learning
Chapter 1: What is Machine Learning?
Defining Machine Learning
Why Learn Machine Learning?
Examples of ML in Everyday Life
Distinguishing ML from Traditional Programming
Types of Machine Learning Systems
The Machine Learning Workflow Overview
Tools You Might Use
Quiz for Chapter 1
Chapter 2: Fundamental Concepts
Data: The Fuel for Machine Learning
Features and Labels Explained
Training, Validation, and Test Sets
Models: Learning from Data
Parameters and Hyperparameters
Introduction to Overfitting and Underfitting
Measuring Performance: Basic Metrics
Quiz for Chapter 2
Chapter 3: Supervised Learning: Regression
Understanding Regression Problems
Introduction to Linear Regression
How Linear Regression Learns
Cost Functions: Measuring Error
Gradient Descent: Finding the Best Fit
Simple Linear Regression Example
Practice: Implementing Simple Linear Regression
Quiz for Chapter 3
Chapter 4: Supervised Learning: Classification
Understanding Classification Problems
Introduction to Logistic Regression
The Sigmoid Function
Decision Boundaries
Introduction to K-Nearest Neighbors (KNN)
How KNN Works
Evaluating Classification Models
Practice: Implementing KNN for Classification
Quiz for Chapter 4
Chapter 5: Unsupervised Learning: Clustering
What is Unsupervised Learning?
Introduction to Clustering
The K-Means Algorithm
Choosing the Number of Clusters (K)
How K-Means Finds Clusters
Limitations of K-Means
Practice: Applying K-Means to Simple Data
Quiz for Chapter 5
Chapter 6: Preparing Your Data
The Importance of Data Preprocessing
Handling Missing Values
Introduction to Feature Scaling
Encoding Categorical Features
Splitting Data into Training and Testing Sets Revisited
Hands-on Practical: Basic Data Cleaning Steps
Quiz for Chapter 6
Chapter 7: Building Your First Machine Learning Model
Recap: The ML Workflow Steps
Choosing the Right Algorithm
Loading and Preparing Data with a Library
Training a Simple Model
Making Predictions
Evaluating Model Performance
Hands-on Practical: End-to-End Simple Model Building
Quiz for Chapter 7

Cost Functions: Measuring Error

Linear regression aims to find the straight line that best represents the relationship within a dataset. But what criterion defines a "best" line? How can we objectively quantify the fit of a particular line to the data points? A method is required to measure the error, often referred to as "cost," associated with any given line. This measurement enables comparison between different potential lines and guides the learning algorithm in adjusting the line for a better fit.

Think about it this way: for any line we draw through our data, some points will be close to the line, and others might be further away. The distance between an actual data point and the point predicted by our line represents an error for that specific prediction.

Calculating the Error for a Single Point

Let's examine a single data point in our training set, represented by its feature(s) xix_ixi​ and its actual target value yiy_iyi​. If our current linear regression model predicts a value y^i\hat{y}_iy^​i​ (pronounced "y-hat") for this input xix_ixi​, the error for this single prediction is simply the difference between the actual value and the predicted value:

Error = Actual Value - Predicted Value Error = yi−y^iy_i - \hat{y}_iyi​−y^​i​

This difference is often called the residual. A positive residual means the prediction was too low, and a negative residual means the prediction was too high. A residual of zero means the prediction was perfect for that data point.

The vertical dashed line shows the error (residual) for one data point: the difference between the actual value (blue dot) and the value predicted by the line (point on the gray line).

Aggregating Errors: The Cost Function

We need a single number that summarizes the total error across all the data points in our training set. Simply summing the individual errors (yi−y^iy_i - \hat{y}_iyi​−y^​i​) isn't very useful, because positive and negative errors could cancel each other out, giving us a misleadingly small total error even if the line is a poor fit.

A common approach is to:

  1. Square each individual error: (yi−y^i)2(y_i - \hat{y}_i)^2(yi​−y^​i​)2. This solves the cancellation problem (squares are always non-negative) and also penalizes larger errors more heavily than smaller ones. An error of 2 becomes 4, while an error of 10 becomes 100.
  2. Calculate the average of these squared errors: Sum up all the squared errors and divide by the number of data points.

This gives us the Mean Squared Error (MSE), a very popular cost function for regression problems.

The formula for MSE is:

J(θ0,θ1)=1N∑i=1N(y^i−yi)2J(\theta_0, \theta_1) = \frac{1}{N} \sum_{i=1}^{N} (\hat{y}_i - y_i)^2J(θ0​,θ1​)=N1​∑i=1N​(y^​i​−yi​)2

Let's break this down:

  • NNN: The total number of data points in our training set.
  • i=1i=1i=1: We start summing from the first data point...
  • ...N... N...N: ...up to the last data point.
  • ∑\sum∑: The summation symbol, meaning we add up the results for each point.
  • yiy_iyi​: The actual target value for the iii-th data point.
  • y^i\hat{y}_iy^​i​: The value predicted by our linear model for the iii-th data point. Remember, for simple linear regression, y^i=θ0+θ1xi\hat{y}_i = \theta_0 + \theta_1 x_iy^​i​=θ0​+θ1​xi​ (where θ0\theta_0θ0​ is the intercept and θ1\theta_1θ1​ is the slope, the parameters our model needs to learn).
  • (y^i−yi)2(\hat{y}_i - y_i)^2(y^​i​−yi​)2: The squared error for the iii-th data point. (Note: Sometimes you'll see (yi−y^i)2(y_i - \hat{y}_i)^2(yi​−y^​i​)2, which gives the same result when squared).
  • 1N\frac{1}{N}N1​: We divide the total sum of squared errors by NNN to get the mean (average).
  • J(θ0,θ1)J(\theta_0, \theta_1)J(θ0​,θ1​): This shows that the MSE is a function of the model parameters (θ0\theta_0θ0​ and θ1\theta_1θ1​). Different values for the slope and intercept will result in a different line, different predictions (y^i\hat{y}_iy^​i​), and therefore a different MSE value.

Sometimes, particularly in statistical contexts or other courses, you might see the formula with 1/(2N)1/(2N)1/(2N) instead of 1/N1/N1/N. The factor of 2 is added for mathematical convenience when calculating derivatives later (specifically for Gradient Descent), but it doesn't change the location of the minimum error. For understanding the concept, 1/N1/N1/N representing the mean is often clearer.

The Goal: Minimizing the Cost

"The Mean Squared Error gives us a single, positive value representing how well our line, defined by specific parameters θ0\theta_0θ0​ and θ1\theta_1θ1​, fits the overall data. A perfect fit would have an MSE of 0 (though this rarely happens with data). A line that fits poorly will have a large MSE."

Therefore, the goal of our learning algorithm is to find the values of θ0\theta_0θ0​ and θ1\theta_1θ1​ that result in the lowest possible MSE.

Minimizing this cost function means finding the line that, on average, makes the smallest squared errors when predicting the target values in our training data. In the next section on Gradient Descent, we'll see how the algorithm systematically adjusts θ0\theta_0θ0​ and θ1\theta_1θ1​ to reduce the value of this cost function, effectively finding the best-fitting line.

  • An Introduction to Statistical Learning: With Applications in R, Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, 2013 (Springer) DOI: 10.1007/978-1-4614-7138-7 - A widely used introductory textbook for statistical learning, providing clear explanations of linear regression, residuals, and the concept of minimizing squared error.
  • CS229 Lecture Notes: Linear Regression, Andrew Ng, Tengyu Ma, 2023 (Stanford University) - Lecture notes from a foundational machine learning course, offering a detailed yet accessible explanation of the linear regression model and the mean squared error cost function.
  • The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Trevor Hastie, Robert Tibshirani, Jerome Friedman, 2009 (Springer) DOI: 10.1007/978-0-387-84858-7 - An authoritative textbook offering a more advanced and comprehensive statistical treatment of linear models and loss functions, including the theoretical underpinnings of MSE.

© 2026 ApX Machine Learning