Method of Least Squares Estimation

Simple linear regression aims to find the best straight line $y = \beta_0 + \beta_1 x$ that describes the relationship between a predictor variable $x$ and a target variable $y$. But what does "best" mean mathematically? A precise method is needed to quantify how well a line fits observed data points $(x_1, y_1), (x_2, y_2), ..., (x_n, y_n)$.

Imagine drawing a potential line through a scatter plot of our data. For each data point $(x_i, y_i)$, the line predicts a value $\hat{y}_i = \beta_0 + \beta_1 x_i$. The difference between the actual observed value $y_i$ and the predicted value $\hat{y}_i$ is called the residual or error for that point:

$e_i = y_i - \hat{y}_i = y_i - (\beta_0 + \beta_1 x_i)$

The residual $e_i$ represents the vertical distance between the actual data point and the line. A good line should have small residuals overall.

Simply summing the residuals ($\sum e_i$) isn't a good measure of fit. Why? Because positive residuals (points above the line) and negative residuals (points below the line) can cancel each other out. A line could have a sum of residuals equal to zero but still be a terrible fit for the data.

To overcome this cancellation issue and ensure that all errors contribute positively to our measure of misfit, we square each residual before summing them up. This gives us the Sum of Squared Residuals (SSR), also known as the Residual Sum of Squares (RSS) or Sum of Squared Errors (SSE):

$SSR = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 = \sum_{i=1}^{n} (y_i - (\beta_0 + \beta_1 x_i))^2$

The Method of Least Squares is the principle that defines the "best" fitting line as the one that minimizes this Sum of Squared Residuals (SSR). We want to find the specific values for the intercept ($\beta_0$) and the slope ($\beta_1$) that make the sum of the squared vertical distances from the data points to the line as small as possible.

How do we find these optimal values, typically denoted as $\hat{\beta}_0$ and $\hat{\beta}_1$? This involves using calculus. We treat the SSR as a function of $\beta_0$ and $\beta_1$. We take the partial derivative of the SSR with respect to $\beta_0$ and the partial derivative with respect to $\beta_1$, set both derivatives equal to zero, and solve the resulting system of two linear equations (often called the "normal equations"). While we won't work through the full derivation here, the solution provides formulas for the estimated coefficients:

The estimated slope ($\hat{\beta}_1$) is calculated as:

$\hat{\beta}_1 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2}$

Here, $\bar{x}$ is the mean of the predictor variable values, and $\bar{y}$ is the mean of the target variable values. Notice that the numerator is proportional to the sample covariance between $x$ and $y$, and the denominator is proportional to the sample variance of $x$. Intuitively, the slope reflects how much $y$ tends to change (covariance) relative to how much $x$ changes (variance).

Once we have the estimated slope $\hat{\beta}_1$, the estimated intercept ($\hat{\beta}_0$) is calculated simply as:

$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$

This formula ensures that the regression line passes through the point $(\bar{x}, \bar{y})$, the center of mass of the data.

Let's visualize this. Imagine the scatter plot of data points. The least squares method finds the unique line that minimizes the sum of the areas of the squares whose sides are the residuals.

Data points (blue), the calculated least squares regression line (red), and the residuals (vertical gray lines). The method minimizes the sum of the areas of the gray squares formed by these residuals.

Calculating these sums and formulas manually can be tedious for large datasets. Fortunately, statistical software and libraries in programming languages like Python handle these computations efficiently. Libraries such as Scikit-learn and Statsmodels provide functions that implement the method of least squares (and more advanced techniques) to estimate regression coefficients directly from your data, allowing you to focus on interpreting the results and evaluating the model. We will see how to use these libraries later in the chapter.