To build effective gradient boosting models, it is helpful to first understand the general principles of ensemble learning. A single predictive model, such as a decision tree, can be prone to high variance or high bias. By combining the predictions of multiple models, we can often achieve a more accurate and generalizable result. This chapter introduces the techniques for combining models.

You will begin by learning what an ensemble method is and how it functions. We will then compare two common strategies: bagging, where models are built independently in parallel, and boosting, where models are built sequentially, with each new model attempting to correct the errors made by the previous ones. This leads to an introduction to the boosting principle and a look at the AdaBoost algorithm, which serves as a direct precursor to the gradient boosting methods we will study later.

Finally, we will define the "weak learners" that act as the components of these ensembles and review how these combination techniques affect the bias-variance tradeoff. The material covered here provides the necessary context for understanding the mechanics of the Gradient Boosting Machine in the next chapter.