Getting Started with Gradient Boosting Algorithms
Chapter 1: Foundations of Ensemble Learning and Boosting
What are Ensemble Methods?
Bagging vs. Boosting
Introduction to the Boosting Principle
The AdaBoost Algorithm: A Precursor to Gradient Boosting
Understanding Weak Learners
Bias-Variance Tradeoff in Ensembles
Chapter 2: The Gradient Boosting Machine (GBM)
From Boosting to Gradient Boosting
The Role of Gradients and Residuals
The GBM Algorithm Step-by-Step
Loss Functions for Regression Tasks
Loss Functions for Classification Tasks
Hands-on Practical: Building a GBM with Python
Chapter 3: Implementing Gradient Boosting with Scikit-Learn
Scikit-Learn's GradientBoostingClassifier
Scikit-Learn's GradientBoostingRegressor
Fitting and Predicting with GBM Models
Interpreting Model Parameters
Feature Importance in GBM
Partial Dependence Plots for Model Interpretation
Hands-on Practical: Building a Predictive Model
Chapter 4: Advanced Gradient Boosting: XGBoost
Why XGBoost? Speed and Performance
Architectural Improvements over Standard GBM
Regularization in XGBoost (L1 and L2)
Handling Missing Values Automatically
Installing and Setting up XGBoost
The XGBoost API: A Walkthrough
Hands-on Practical: Training an XGBoost Model
Chapter 5: Advanced Gradient Boosting: LightGBM and CatBoost
Introduction to LightGBM: Gradient-based One-Side Sampling
LightGBM's Exclusive Feature Bundling
Introduction to CatBoost: Handling Categorical Features
CatBoost's Ordered Boosting and Symmetric Trees
Performance Comparison: XGBoost vs. LightGBM vs. CatBoost
Hands-on Practical: Implementing LightGBM and CatBoost
Chapter 6: Hyperparameter Tuning and Model Optimization
The Importance of Hyperparameter Tuning
Principal Hyperparameters in Gradient Boosting
Tuning the Number of Estimators and Learning Rate
Controlling Tree Complexity
Subsampling Parameters for Regularization
A Structured Approach to Tuning
Using Grid Search and Randomized Search
Hands-on Practical: Optimizing a Gradient Boosting Model
Performance Comparison: XGBoost vs. LightGBM vs. CatBoost
Was this section helpful?
- XGBoost: A Scalable Tree Boosting System, Tianqi Chen and Carlos Guestrin, 2016 Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16) (Association for Computing Machinery (ACM)) DOI: 10.1145/2939672.2939785 - The foundational paper introducing XGBoost, detailing its parallel and distributed training, cache-aware access, and tree boosting algorithms.
- LightGBM: A Highly Efficient Gradient Boosting Decision Tree, Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu, 2017 Advances in Neural Information Processing Systems 30 (NeurIPS 2017) (Curran Associates, Inc.) DOI: 10.5555/3295222.3295232 - Presents LightGBM, outlining its Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB) techniques for faster training and leaf-wise tree growth.
- CatBoost: Gradient Boosting with Decision Trees, Anna Veronika Dorogush, Andrey Gulin, Gleb Gusev, Neliya Kazimov, and Liudmila Prokhorenkova, 2020 Journal of Machine Learning Research, Vol. 21 - Describes the CatBoost algorithm, with a focus on its ordered boosting scheme for categorical features and symmetric (oblivious) tree construction.
- XGBoost Documentation, XGBoost Contributors, 2024 - The official documentation providing practical guidance, API details, and tutorials for using XGBoost.