Getting Started with Gradient Boosting Algorithms
Chapter 1: Foundations of Ensemble Learning and Boosting
What are Ensemble Methods?
Bagging vs. Boosting
Introduction to the Boosting Principle
The AdaBoost Algorithm: A Precursor to Gradient Boosting
Understanding Weak Learners
Bias-Variance Tradeoff in Ensembles
Chapter 2: The Gradient Boosting Machine (GBM)
From Boosting to Gradient Boosting
The Role of Gradients and Residuals
The GBM Algorithm Step-by-Step
Loss Functions for Regression Tasks
Loss Functions for Classification Tasks
Hands-on Practical: Building a GBM with Python
Chapter 3: Implementing Gradient Boosting with Scikit-Learn
Scikit-Learn's GradientBoostingClassifier
Scikit-Learn's GradientBoostingRegressor
Fitting and Predicting with GBM Models
Interpreting Model Parameters
Feature Importance in GBM
Partial Dependence Plots for Model Interpretation
Hands-on Practical: Building a Predictive Model
Chapter 4: Advanced Gradient Boosting: XGBoost
Why XGBoost? Speed and Performance
Architectural Improvements over Standard GBM
Regularization in XGBoost (L1 and L2)
Handling Missing Values Automatically
Installing and Setting up XGBoost
The XGBoost API: A Walkthrough
Hands-on Practical: Training an XGBoost Model
Chapter 5: Advanced Gradient Boosting: LightGBM and CatBoost
Introduction to LightGBM: Gradient-based One-Side Sampling
LightGBM's Exclusive Feature Bundling
Introduction to CatBoost: Handling Categorical Features
CatBoost's Ordered Boosting and Symmetric Trees
Performance Comparison: XGBoost vs. LightGBM vs. CatBoost
Hands-on Practical: Implementing LightGBM and CatBoost
Chapter 6: Hyperparameter Tuning and Model Optimization
The Importance of Hyperparameter Tuning
Principal Hyperparameters in Gradient Boosting
Tuning the Number of Estimators and Learning Rate
Controlling Tree Complexity
Subsampling Parameters for Regularization
A Structured Approach to Tuning
Using Grid Search and Randomized Search
Hands-on Practical: Optimizing a Gradient Boosting Model
CatBoost's Ordered Boosting and Symmetric Trees
Was this section helpful?
- CatBoost: Unbiased Gradient Boosting with Categorical Features, Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, Andrey Gulin, 2018 Advances in Neural Information Processing Systems 31 (NeurIPS) DOI: 10.55986/neurips.2018.0cce69792070659637c38bc8f7c6530a - This foundational paper introduces the CatBoost algorithm, detailing its core innovations including Ordered Boosting to combat target leakage and the use of symmetric (oblivious) decision trees for efficient prediction and regularization.
- CatBoost Documentation, Yandex, 2023 - The official documentation provides comprehensive guides, practical examples, and explanations of CatBoost features, including Ordered Boosting and symmetric trees.
- The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Trevor Hastie, Robert Tibshirani, Jerome Friedman, 2009 (Springer) - A classic textbook providing a foundational understanding of machine learning algorithms, including decision trees, boosting, and ensemble methods, which provides context for the architectural choices in CatBoost regarding regularization and model complexity.