Shrinkage as Implicit Regularization

The model is built sequentially in the gradient boosting framework:

$$F_m(x) = F_{m-1}(x) + \nu h_m(x)$$

Here, $F_{m-1}(x)$ is the model after $m-1$ boosting rounds, $h_m(x)$ is the new base learner (typically a decision tree) trained to fit the residual errors (or gradients) from the previous stage, and $\nu$ (often denoted as eta or learning_rate in libraries) is the shrinkage parameter, typically a small number between 0 and 1 (e.g., 0.01, 0.1).

At first glance, $\nu$ looks identical to the learning rate used in standard gradient descent optimization algorithms. Its role here is analogous but operates within the context of function space. Shrinkage scales the contribution of each new tree added to the ensemble. By setting $\nu < 1$, we deliberately slow down the learning process. Instead of allowing each new tree $h_m(x)$ to fully correct the errors of the previous model $F_{m-1}(x)$, we only add a fraction $\nu$ of its prediction.

Why is this beneficial for regularization?

1. Reduces Influence of Individual Trees: A small $\nu$ diminishes the impact of each individual tree. This prevents any single tree, which might capture noise or specific patterns only present in the training data, from having an outsized effect on the final prediction. The model becomes less sensitive to the peculiarities of individual base learners.
2. Requires More Trees: Slowing down the learning process means that more boosting rounds (i.e., more trees, $M$) are generally needed to achieve a similar level of fit on the training data compared to a model trained with $\nu=1$.
3. Improves Generalization through Averaging: The need for more trees has a positive side effect. The final prediction $F_M(x)$ becomes an aggregation over a larger number of diverse base learners. Each tree focuses on the remaining errors, and by combining many slightly different perspectives (each weighted less heavily due to $\nu$), the model tends to capture the underlying signal more effectively while averaging out the noise. This ensemble effect, encouraged by shrinkage, leads to better generalization performance on unseen data.

Think of it like taking smaller, more cautious steps down the optimization path in function space. Larger steps (high $\nu$) might quickly reduce the training error but risk overshooting the optimal functional fit or fitting noise aggressively. Smaller steps (low $\nu$) proceed more slowly but allow the model to gradually refine its predictions, integrating information from many trees and resulting in a smoother, more generalizable final function.

This regularization effect is considered implicit because shrinkage doesn't add an explicit penalty term to the loss function based on model complexity (like L1/L2 regularization does) nor does it directly constrain the structure of the trees (like setting max_depth). Instead, it modifies the boosting process itself, inherently encouraging solutions that rely on the collaboration of many weak learners.

The following chart illustrates this concept. Notice how a lower learning rate ($\nu = 0.1$) leads to slower convergence on the training set but achieves a better (lower) validation error compared to a higher learning rate ($\nu = 0.8$), which overfits rapidly.

Training and validation error curves for models trained with high ($\nu=0.8$) and low ($\nu=0.1$) shrinkage rates. The lower rate requires more rounds but results in better validation performance, mitigating overfitting.

In practice, shrinkage is almost always used (values far less than 1.0 are typical). It forms a fundamental trade-off with the number of boosting rounds ($M$). A common strategy is to set $\nu$ to a small value (e.g., 0.01 to 0.1) and then determine the optimal $M$ using a validation set, often employing early stopping (discussed later in this chapter). While very small $\nu$ values can significantly increase computation time due to the large $M$ required, the resulting generalization improvement often justifies the cost. Shrinkage works with other regularization techniques like tree constraints and subsampling to build gradient boosting models.