The Simple Linear Regression Model

Regression analysis helps quantify and understand relationships between variables. Simple linear regression (SLR) specifically models how a single dependent or response variable changes in response to another single independent or predictor variable.

Imagine you have data on two variables, say, years of experience ($x$) and salary ($y$). You might suspect that as experience increases, salary tends to increase as well. Simple linear regression provides a formal way to model this suspected linear relationship.

The Underlying Model

"At its core, simple linear regression assumes that the relationship between the independent variable $x$ and the dependent variable $y$ can be approximated by a straight line. However, data rarely falls perfectly on a line. There's almost always some scatter or variability. To account for this, the theoretical model for simple linear regression is written as:"

$y = \beta_0 + \beta_1 x + \epsilon$

Let's break down this equation:

• $y$: This is the dependent variable, the outcome we are trying to predict or explain (e.g., salary).
• $x$: This is the independent variable, the predictor we are using to explain $y$ (e.g., years of experience).
• $\beta_0$ (Beta-naught): This is the intercept parameter. It represents the theoretical average value of $y$ when $x$ is equal to zero. In some contexts, the intercept might not have a direct practical interpretation (e.g., zero years of experience might not be meaningful or present in the data), but it's mathematically necessary to position the line correctly.
• $\beta_1$ (Beta-one): This is the slope parameter. It represents the theoretical change in the average value of $y$ for a one-unit increase in $x$. If $\beta_1$ is positive, it indicates a positive linear relationship (as $x$ increases, $y$ tends to increase). If $\beta_1$ is negative, it indicates a negative linear relationship (as $x$ increases, $y$ tends to decrease). If $\beta_1$ is zero, it suggests no linear relationship between $x$ and $y$. The magnitude of $\beta_1$ indicates the strength of the linear association.
• $\epsilon$ (Epsilon): This is the error term or residual. It represents the difference between the actual observed value of $y$ and the value predicted by the linear relationship ($\beta_0 + \beta_1 x$). This term captures all the variability in $y$ that is not explained by the linear relationship with $x$. This could be due to other unmeasured factors influencing $y$, inherent randomness, or measurement error. We typically make assumptions about these errors, most commonly that they are independent, have a mean of zero, and possess a constant variance ($\sigma^2$).

Think of the $\beta_0 + \beta_1 x$ part as the deterministic, linear component of the relationship, and $\epsilon$ as the random, unexplained component.

From Population to Sample

The equation $y = \beta_0 + \beta_1 x + \epsilon$ describes the theoretical relationship in the entire population. In practice, we rarely have access to population data. Instead, we work with a sample drawn from the population. Our goal is to use the sample data to estimate the unknown population parameters $\beta_0$ and $\beta_1$.

We denote the estimates calculated from the sample data as $b_0$ (or sometimes $\hat{\beta}_0$) and $b_1$ (or $\hat{\beta}_1$). The estimated regression line based on the sample is then:

$\hat{y} = b_0 + b_1 x$

Here, $\hat{y}$ (read "y-hat") represents the predicted value of $y$ for a given value of $x$, based on our sample estimates. The difference between an observed value $y_i$ in our sample and its corresponding predicted value $\hat{y}_i$ is the sample residual, $e_i = y_i - \hat{y}_i$. These sample residuals $e_i$ are our observable stand-ins for the unobservable theoretical errors $\epsilon_i$.

The challenge, which we'll address in the next section, is how to find the "best" values for $b_0$ and $b_1$ based on our sample data points $(x_i, y_i)$.

Visualizing the Linear Relationship

A scatter plot is the ideal way to visualize the data and the potential linear relationship before fitting a model. Simple linear regression essentially tries to find the line that best cuts through the cloud of points on this scatter plot.

A scatter plot showing individual data points (blue dots) and a potential line (red dashed) representing the simple linear regression model $\hat{y} = b_0 + b_1 x$. The goal is to find the line that minimizes the overall distance between the line and the points.

Understanding this basic model structure is fundamental. It not only allows us to model simple relationships but also serves as the foundation for more complex regression techniques, including multiple linear regression (using multiple predictors) and polynomial regression (modeling curves), which are frequently used in machine learning.