Having reviewed ensemble methods and the role of decision trees, let's formalize the sequential learning process that underpins boosting algorithms. This process is known as additive modeling. Unlike methods like Bagging, where base learners are often trained independently and in parallel, additive models are built iteratively. Each new component added to the model focuses on the shortcomings of the existing ensemble.

The Sequential Nature

Imagine building a predictive model step-by-step. You start with an initial, often very simple, prediction. Then, you assess where this initial prediction falls short. Based on this assessment, you add a new component, a simple model, specifically designed to compensate for the errors made so far. You repeat this process, incrementally refining the overall model by adding components that address the remaining errors.

This iterative refinement is the essence of additive modeling. The final prediction is the sum (or addition) of the predictions from all the components built sequentially.

Formalizing the Framework

Mathematically, an additive model $F_M(x)$ making predictions for an input $x$ after $M$ stages (or iterations) can be represented as:

F_M(x) = F_0(x) + \sum_{m=1}^{M} \beta_m h_m(x)

Let's break down this equation:

$F_M(x)$ : The final prediction of the ensemble after $M$ steps.
$F_0(x)$ : An initial base model or starting prediction. For many problems, this is simply the mean (for regression) or log-odds (for classification) of the target variable in the training set. It represents our best guess before adding any sophisticated learners.
$h_m(x)$ : The $m$ -th base learner (often a decision tree) added at step $m$ . This learner is trained specifically to address the errors or deficiencies of the model built up to step $m-1$ , denoted as $F_{m-1}(x)$ .
$\beta_m$ : A coefficient or weight applied to the prediction of the $m$ -th base learner. This term controls the contribution of each new learner to the overall ensemble. In gradient boosting, this is closely related to the learning rate or shrinkage parameter, which helps prevent overfitting by dampening the impact of each individual step.
$M$ : The total number of base learners (or boosting stages).

The Iterative Process

The model construction proceeds as follows:

Initialization: Start with an initial model $F_0(x)$ . This is often a constant value that minimizes the loss function over the training data (e.g., the mean for squared error loss).
Iteration $m=1$ to $M$ :
- Compute the errors (residuals or gradients, as we'll see later) made by the current ensemble $F_{m-1}(x)$ . These errors represent the "unexplained" part of the target variable.
- Train a new base learner $h_m(x)$ to predict these errors from the previous step. The goal of $h_m(x)$ is to capture the patterns in the remaining errors.
- Determine the optimal coefficient $\beta_m$ for this new learner (often involves minimizing the overall loss function).
- Update the ensemble model: $F_m(x) = F_{m-1}(x) + \beta_m h_m(x)$ .

This iterative process is visualized below:

The additive modeling process: Start with an initial model ( $F_0$ ), calculate errors, fit a new base learner ( $h_m$ ) to those errors, and add it to the ensemble ( $F_m = F_{m-1} + \beta_m h_m$ ). Repeat for $M$ steps.

Why Additive Modeling?

The power of this framework lies in its flexibility and focus. By concentrating on the errors of the preceding model, each new learner tackles the aspects of the problem that the current ensemble finds most difficult. This allows the model to gradually improve performance, potentially capturing complex patterns that a single model might miss.

Gradient Boosting is a highly successful algorithm family that operates within this additive modeling framework. It provides a specific, mathematically grounded way to determine how each new base learner $h_m(x)$ should be trained to best correct the errors of $F_{m-1}(x)$ , using the concept of gradient descent in function space. We will explore this connection in detail as we move forward. Understanding this additive structure is foundational to grasping the mechanics of GBM, XGBoost, LightGBM, and CatBoost.