This chapter revisits fundamental concepts that form the bedrock upon which advanced gradient boosting algorithms are built. To understand the mechanics and optimizations of specific implementations like XGBoost, LightGBM, and CatBoost, we first need a clear grasp of their origins and the core ideas they evolved from.

We will begin with a brief review of ensemble methods, focusing on how boosting differs from techniques like bagging. We'll then examine decision trees, the typical base learners used within boosting frameworks, and discuss the properties that make them suitable. Following this, we revisit the additive modeling approach, where models are constructed sequentially, with each new model attempting to correct the errors of the previous ones.

Understanding how these models learn involves optimization, so we will touch upon the fundamentals of gradient descent as it applies in this context. Finally, we will introduce the standard Gradient Boosting Machine (GBM) algorithm itself, setting the stage for analyzing its more advanced successors in subsequent chapters. This review ensures everyone has the necessary background before proceeding to more complex material.