Review of Common Supervised Learning Algorithms

Supervised learning algorithms are central to building predictive models. These algorithms aim to construct a model that learns a mapping from input features (denoted as $X$) to an output target variable ($y$). This learning process is based on a collection of example pairs $(X, y)$, with the 'supervision' arising from the known output labels in the training data.

Supervised learning problems generally fall into two main categories:

1. Regression: Predicting a continuous numerical value. Examples include predicting house prices, stock values, or temperature.
2. Classification: Predicting a discrete category or class label. Examples include classifying emails as spam or not spam, identifying images of cats vs. dogs, or predicting customer churn (yes/no).

This chapter focuses on implementing and tuning models for both types of problems. Here's a refresher on the algorithms mentioned:

Linear Models

Linear models are fundamental in machine learning due to their simplicity and interpretability. They model the target variable as a linear combination of the input features.

Linear Regression

This is the go-to algorithm for many regression tasks. It assumes a linear relationship between the input features ($X$) and the continuous target variable ($y$). The model aims to find the best-fitting straight line (or hyperplane in higher dimensions) through the data points.

The relationship is modeled by the equation:

$$y \approx \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_n x_n$$

Here, $y$ is the predicted value, $x_1, \dots, x_n$ are the input features, and $\beta_1, \dots, \beta_n$ are the coefficients (or weights) learned by the model, representing the change in $y$ for a one-unit change in the corresponding feature, holding others constant. $\beta_0$ is the intercept term, representing the expected value of $y$ when all features are zero.

Training involves finding the coefficients $\beta$ that minimize a cost function, typically the sum of squared differences between the predicted values and the actual values (Residual Sum of Squares or RSS). While straightforward, its effectiveness relies on certain assumptions, such as linearity between features and target, and independence of errors.

Logistic Regression

Despite its name, Logistic Regression is used for classification tasks, primarily binary classification (two classes, e.g., 0 or 1, Yes or No). It models the probability that an input $X$ belongs to a particular class.

It adapts linear regression by feeding the linear combination of inputs through a sigmoid (or logistic) function:

$$P(y=1 | X) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \dots + \beta_n x_n)}}$$

The sigmoid function squashes the output of the linear equation into the range $[0, 1]$, allowing it to be interpreted as a probability. A threshold (often 0.5) is then used to convert this probability into a class prediction: if $P(y=1|X) > 0.5$, predict class 1, otherwise predict class 0.

Training involves finding the coefficients $\beta$ that maximize the likelihood of observing the actual class labels in the training data, often using techniques like gradient descent. It provides interpretable coefficients related to the log-odds of the outcome.

Tree-Based Models

Tree-based models partition the feature space into a set of rectangles and fit a simple model (like a constant) in each one. They are versatile and can handle both regression and classification.

Decision Trees

A Decision Tree builds a model in the form of a tree structure. It breaks down a dataset into smaller and smaller subsets while incrementally developing an associated decision tree.

The process starts at the root node and recursively splits the data based on feature values that best separate the target variable. Common splitting criteria include Gini impurity or information gain (entropy) for classification, and variance reduction for regression. The splits continue until a stopping criterion is met (e.g., maximum depth, minimum samples per leaf). The terminal nodes are called leaf nodes and contain the predictions (the majority class for classification, or the average value for regression).

A simple representation of a decision tree structure for classification. Each internal node tests a feature, branches represent the outcome of the test, and leaf nodes assign a class label.

Decision trees are relatively easy to understand and visualize. However, single decision trees can be prone to overfitting, meaning they learn the training data too well, including its noise, and may not generalize well to new, unseen data.

Random Forests

Random Forests address the overfitting issue of single decision trees by building an ensemble of multiple decision trees. It's a bagging method (Bootstrap Aggregating) with an additional layer of randomness.

Here's the idea:

1. Bootstrap Sampling: Create multiple different training datasets by sampling with replacement from the original dataset.
2. Feature Randomness: For each tree, at each split point, use only a random subset of features rather than all available features.
3. Build Trees: Train a decision tree on each bootstrapped dataset using the randomized feature subsets. These trees are typically grown deep.
4. Aggregate Predictions: For regression, average the predictions of all trees. For classification, use majority voting (the most predicted class across all trees).

This combination of bootstrapping and feature randomness helps to decorrelate the individual trees. The ensemble model is generally more accurate than any single tree, significantly reducing variance and overfitting. The trade-off is a loss in direct interpretability compared to a single decision tree.

Gradient Boosting Machines (GBM)

Gradient Boosting is another powerful ensemble technique that builds models sequentially, with each new model attempting to correct the errors made by the previous ones. Unlike Random Forests where trees are built independently, GBMs build trees additively.

The core concept involves:

1. Start with a simple initial model (e.g., predicting the mean value for regression, or log-odds for classification).
2. Iteratively add new trees to the ensemble.
3. Each new tree is trained to predict the residual errors (the difference between the actual values and the current ensemble's predictions) of the preceding model.
4. The predictions of these new trees are added to the ensemble prediction, usually scaled by a learning rate (a small step size) to prevent overfitting.

This sequential, error-correcting process allows GBMs to fit the data very effectively and often achieve high predictive accuracy. Popular and efficient implementations include:

• XGBoost (Extreme Gradient Boosting): Known for its speed, regularization options (L1 and L2) to combat overfitting, built-in handling of missing values, and cross-validation capabilities.
• LightGBM (Light Gradient Boosting Machine): Also focuses on performance, using techniques like gradient-based one-side sampling (GOSS) and exclusive feature bundling (EFB) to speed up training, especially on large datasets. It often uses leaf-wise tree growth instead of level-wise.

"These algorithms are frequently used in machine learning competitions and applications due to their performance. However, they have more hyperparameters to tune compared to simpler models like linear regression or single decision trees."

This review provides the foundation. In the following sections, we will move into the practical implementation, evaluation, and tuning of these supervised learning models using scikit-learn and other relevant Python libraries.