In the previous chapter, we examined the principle of boosting using AdaBoost, where models are built sequentially to correct the mistakes of their predecessors. This chapter builds on that by introducing the Gradient Boosting Machine (GBM), a more generalized and flexible boosting algorithm.
Instead of adjusting weights on training samples, Gradient Boosting frames the problem as a functional gradient descent. Each new weak learner is not trained on the original target but on the negative gradient of the loss function with respect to the previous model's output. For a common regression problem using mean squared error, this gradient simplifies to the residual error, or . This approach allows Gradient Boosting to be used with any differentiable loss function, making it applicable to a wide variety of tasks.
Throughout this chapter, we will break down the mechanics of the GBM algorithm. You will learn:
To solidify these topics, the chapter concludes with a practical exercise where you will implement a basic version of the GBM algorithm from scratch using Python.
2.1 From Boosting to Gradient Boosting
2.2 The Role of Gradients and Residuals
2.3 The GBM Algorithm Step-by-Step
2.4 Loss Functions for Regression Tasks
2.5 Loss Functions for Classification Tasks
2.6 Hands-on Practical: Building a GBM with Python
© 2026 ApX Machine LearningEngineered with