Linear Regression Fundamentals

Linear regression serves as a foundational algorithm in supervised learning for regression problems. Its primary aim is to model the relationship between one or more explanatory variables (features) and a continuous target variable by fitting a linear equation to the observed data. Think of it as finding the "best-fitting" straight line (or hyperplane in higher dimensions) through your data points.

Simple Linear Regression: Modeling with One Feature

The simplest form is simple linear regression, where we predict a target variable $y$ based on a single feature $x$. The relationship is modeled by the equation of a straight line:

$$y = \beta_0 + \beta_1 x + \epsilon$$

Let's break down this equation:

• $y$: The target variable we want to predict (e.g., house price).
• $x$: The input feature we are using for prediction (e.g., house size in square feet).
• $\beta_0$: The intercept (or bias). This is the predicted value of $y$ when $x$ is zero. Geometrically, it's where the regression line crosses the y-axis.
• $\beta_1$: The coefficient (or slope) for the feature $x$. It represents the change in the target variable $y$ for a one-unit increase in the feature $x$. A positive $\beta_1$ means $y$ tends to increase as $x$ increases, while a negative $\beta_1$ indicates an inverse relationship.
• $\epsilon$: The error term (or residual). This represents the difference between the actual observed value of $y$ and the value predicted by the model ($\beta_0 + \beta_1 x$). It captures variability in $y$ not explained by the linear relationship with $x$.

The goal of training a simple linear regression model is to find the optimal values for the intercept ($\beta_0$) and the slope ($\beta_1$) that minimize the overall error ($\epsilon$) across all data points.

A scatter plot showing data points and the best-fitting straight line found by simple linear regression.

Multiple Linear Regression: Modeling with Multiple Features

"In most scenarios, the target variable depends on more than one feature. Multiple linear regression extends the simple case by considering multiple input features ($x_1, x_2, ..., x_n$) to predict the target $y$."

The equation becomes:

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_n x_n + \epsilon$$

Here:

• $y$: The target variable.
• $x_1, x_2, \dots, x_n$: The input features (e.g., house size, number of bedrooms, age of the house).
• $\beta_0$: The intercept, representing the predicted value of $y$ when all features ($x_1, \dots, x_n$) are zero.
• $\beta_1, \beta_2, \dots, \beta_n$: The coefficients for each corresponding feature. Each $\beta_i$ represents the expected change in $y$ for a one-unit increase in feature $x_i$, assuming all other features are held constant. This "holding constant" aspect is important for interpretation.
• $\epsilon$: The error term, capturing the variability not explained by the linear combination of features.

The model finds the best-fitting hyperplane in the n-dimensional feature space. The objective remains the same: find the coefficients ($\beta_0, \beta_1, \dots, \beta_n$) that minimize the prediction errors.

The "Linear" Assumption

It's important to recognize that linear regression, by its nature, assumes a linear relationship between the features and the target variable. The model tries to capture the underlying trend using a straight line or a hyperplane. If the true relationship is highly non-linear (e.g., curves sharply), a simple linear model might not provide accurate predictions (it might underfit the data). We will explore ways to handle non-linearity later, but for many problems, linear regression provides a good starting point and a highly interpretable model.

How the "Best Fit" is Found

How does the algorithm determine the optimal values for the coefficients ($\beta$s)? The most common method is Ordinary Least Squares (OLS). OLS works by finding the line (or hyperplane) that minimizes the sum of the squared differences between the actual target values ($y$) and the values predicted by the model ($\hat{y}$). Squaring the differences ensures that positive and negative errors don't cancel each other out and penalizes larger errors more heavily. While the mathematical details involve calculus or linear algebra, the goal is straightforward: minimize the total squared error.

Interpretability

One of the significant advantages of linear regression is its interpretability. The learned coefficients ($\beta_1, \dots, \beta_n$) directly quantify the relationship between each feature and the target, assuming other features are constant. For instance, if $\beta_1$ for 'house size' is 150, it suggests that, on average, each additional square foot is associated with a $150 increase in house price, holding other factors like the number of bedrooms constant. This makes it easier to understand the model's behavior and explain its predictions.

Now that we have a grasp of the concepts behind linear regression, the next sections will demonstrate how to implement, train, and evaluate these models using Scikit-learn.