In the previous chapter, we examined the principle of boosting using AdaBoost, where models are built sequentially to correct the mistakes of their predecessors. This chapter builds on that by introducing the Gradient Boosting Machine (GBM), a more generalized and flexible boosting algorithm.

Instead of adjusting weights on training samples, Gradient Boosting frames the problem as a functional gradient descent. Each new weak learner is not trained on the original target but on the negative gradient of the loss function with respect to the previous model's output. For a common regression problem using mean squared error, this gradient simplifies to the residual error, or $y - \hat{y}$. This approach allows Gradient Boosting to be used with any differentiable loss function, making it applicable to a wide variety of tasks.

Throughout this chapter, we will break down the mechanics of the GBM algorithm. You will learn:

• How Gradient Boosting generalizes the boosting framework.
• The function of gradients and their connection to residuals.
• The step-by-step procedure for building a GBM.
• Common loss functions for both regression and classification.

To solidify these topics, the chapter concludes with a practical exercise where you will implement a basic version of the GBM algorithm from scratch using Python.