ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Mastering Gradient Boosting Algorithms
Chapter 1: Gradient Boosting Foundations Revisited
Ensemble Methods: A Recap
Decision Trees as Base Learners
The Additive Modeling Framework
Gradient Descent Fundamentals
Introducing the Gradient Boosting Machine (GBM)
Chapter 2: The Gradient Boosting Algorithm in Depth
Functional Gradient Descent
Deriving the Generic GBM Algorithm
Common Loss Functions for Regression
Common Loss Functions for Classification
The Role of Shrinkage (Learning Rate)
Subsampling Techniques (Stochastic Gradient Boosting)
Implementing GBM with Scikit-learn
Practice: Building a Basic GBM Model
Chapter 3: Regularization in Gradient Boosting
Overfitting Challenges in Boosting
Tree Constraints: Depth, Nodes, and Splits
Shrinkage as Implicit Regularization
Subsampling (Stochastic Gradient Boosting)
Regularized Objective Functions (L1/L2)
Early Stopping Strategies
Hands-on Practical: Applying Regularization
Chapter 4: XGBoost: Extreme Gradient Boosting
Motivation and Enhancements over GBM
The Regularized Learning Objective
Split Finding Algorithm: Exact Greedy
Split Finding Algorithm: Approximate Greedy
Sparsity-Aware Split Finding
System Optimizations: Cache Awareness and Parallelism
XGBoost API: Parameters and Configuration
Hands-on Practical: Implementing XGBoost
Chapter 5: LightGBM: Light Gradient Boosting Machine
Motivation: Addressing XGBoost's Limitations
Gradient-based One-Side Sampling (GOSS)
Exclusive Feature Bundling (EFB)
Histogram-Based Split Finding
Leaf-Wise Tree Growth
Optimized Categorical Feature Handling
LightGBM API: Parameters and Configuration
Hands-on Practical: Implementing LightGBM
Chapter 6: CatBoost: Gradient Boosting on Decision Trees
Motivation: Challenges with Categorical Data
Ordered Target Statistics (Ordered TS)
Addressing Prediction Shift: Ordered Boosting
Handling Feature Combinations
Oblivious Trees
GPU Training Acceleration
CatBoost API: Parameters and Configuration
Hands-on Practical: Implementing CatBoost
Chapter 7: Advanced Topics and Customization
Model Interpretability with SHAP
TreeSHAP for Gradient Boosting Models
Global vs. Local Explanations
Probability Calibration for Classification
Implementing Custom Loss Functions
Implementing Custom Evaluation Metrics
Handling Imbalanced Datasets with Boosting
Practice: Custom Objectives and SHAP
Chapter 8: Hyperparameter Optimization Strategies
The Importance of Hyperparameter Tuning
Identifying Critical Hyperparameters
Systematic Tuning: Grid Search and Randomized Search
Advanced Tuning: Bayesian Optimization
Hyperparameter Optimization Frameworks (Optuna, Hyperopt)
Tuning Strategy: From Coarse to Fine
Cross-Validation Strategies for Tuning
Hands-on Practical: Advanced Tuning with Optuna
Chapter 9: Gradient Boosting for Specialized Tasks
Learning to Rank with Gradient Boosting
Ranking Objective Functions (Pairwise, Listwise)
Survival Analysis with Gradient Boosting
Survival Objective Functions (Cox PH)
Quantile Regression with Gradient Boosting
Quantile Loss Function Implementation
Multi-Output Gradient Boosting
Practice: Implementing Ranking with XGBoost

Probability Calibration for Classification

While gradient boosting models often achieve high predictive accuracy according to metrics like AUC or F1-score, their raw output scores for classification tasks might not directly correspond to true probabilities. For instance, a model predicting a score of 0.9 for a class doesn't necessarily mean there's a 90% objective probability that the instance belongs to that class. The optimization process, particularly with loss functions like log loss, pushes scores towards 0 and 1 to improve discrimination, but this can distort the probabilistic interpretation. When the actual probability estimates are important for decision-making, risk assessment, or downstream tasks like model stacking, calibrating these outputs becomes a necessary step.

Understanding Probability Calibration

Probability calibration is the process of transforming the raw output scores of a classifier into probabilities that better reflect the true likelihood of outcomes. A perfectly calibrated binary classifier has the property that, among the instances where it predicts a probability ppp, the actual fraction of positive instances is indeed close to ppp.

A common way to visualize calibration is through a reliability diagram (also known as a calibration curve). This plot bins the predicted probabilities (e.g., 0-0.1, 0.1-0.2, ..., 0.9-1.0) and, for each bin, plots the average predicted probability against the actual fraction of positive instances within that bin.

The diagonal dashed line represents perfect calibration. Bars below the diagonal indicate over-prediction (predicted probability is higher than the actual frequency), while bars above indicate under-prediction. The red bars show an uncalibrated model, while the blue bars show improved calibration after applying a technique.

Boosting algorithms, while powerful, can produce uncalibrated probabilities. The additive nature, focus on correcting errors (often leading to extreme scores for confidently classified points), and the specifics of tree construction can contribute to this. Therefore, checking and potentially correcting the calibration of XGBoost, LightGBM, or CatBoost classifiers is often a good practice.

Calibration Methods

Two widely used methods for probability calibration are Platt Scaling and Isotonic Regression. Both are typically applied after the main classifier has been trained, using a separate calibration dataset (e.g., a hold-out validation set) that was not used for training the original model. Using the training data for calibration would lead to overly optimistic results.

Platt Scaling

Platt Scaling assumes that the distortion between the model's output scores and the true probabilities can be corrected by fitting a sigmoid function. For a binary classifier outputting scores sss, Platt Scaling finds parameters AAA and BBB such that the calibrated probability P(class=1∣s)P(\text{class}=1 | s)P(class=1∣s) is estimated by:

PPlatt(y=1∣s)=11+exp⁡(As+B)P_{\text{Platt}}(y=1|s) = \frac{1}{1 + \exp(A s + B)}PPlatt​(y=1∣s)=1+exp(As+B)1​

The parameters AAA and BBB are typically found by optimizing the log loss (or cross-entropy) on the calibration set, relating the original model's scores sss to the true labels yyy in that set.

Platt Scaling works best when the calibration curve is monotonically increasing and has a sigmoid shape. It's computationally efficient and requires relatively less data compared to Isotonic Regression.

Isotonic Regression

Isotonic Regression is a non-parametric approach. It fits a non-decreasing, piecewise-constant function to the relationship between the model's predicted scores and the observed target values in the calibration set. It finds the best-fitting (in a least-squares sense) step function that preserves the order of the inputs.

The primary algorithm used is Pool Adjacent Violators Algorithm (PAVA). Isotonic Regression makes fewer assumptions about the shape of the distortion compared to Platt Scaling. If the calibration curve is not sigmoid-shaped, Isotonic Regression can often provide a better fit. However, it generally requires more data to yield stable results and can sometimes produce sharp steps in the calibrated probabilities, which might not always be desirable.

Implementation with Scikit-learn

Scikit-learn provides a convenient wrapper, CalibratedClassifierCV, to handle both model training and calibration. It can perform calibration using either Platt Scaling (method='sigmoid') or Isotonic Regression (method='isotonic').

Internally, CalibratedClassifierCV uses cross-validation. For each fold:

  1. A base classifier (e.g., XGBoost, LightGBM) is trained on the training part of the fold.
  2. This classifier makes predictions on the testing part of the fold.
  3. A calibrator (Sigmoid or Isotonic) is trained using these predictions and the true labels from the testing part.

Finally, the base classifier is retrained on all the data, and the calibrators learned during cross-validation are averaged (or selected) to form the final calibrator.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.calibration import CalibratedClassifierCV, CalibrationDisplay
from xgboost import XGBClassifier
import matplotlib.pyplot as plt

# Generate synthetic data
X, y = make_classification(n_samples=1000, n_features=20, n_informative=10, n_redundant=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
X_train, X_calib, y_train, y_calib = train_test_split(X_train, y_train, test_size=0.3, random_state=42) # Further split for explicit calibration set if not using CV


# 1. Train the base XGBoost model (example)
base_model = XGBClassifier(use_label_encoder=False, eval_metric='logloss', random_state=42)
base_model.fit(X_train, y_train)
y_pred_proba_uncalibrated = base_model.predict_proba(X_test)[:, 1]

# 2. Calibrate using CalibratedClassifierCV (handles CV internally)
# Platt Scaling
calibrated_sigmoid = CalibratedClassifierCV(
    base_estimator=base_model,
    method='sigmoid',
    cv=3 # Use 3-fold cross-validation
)
calibrated_sigmoid.fit(X_train, y_train) # Fit on the original training set
y_pred_proba_sigmoid = calibrated_sigmoid.predict_proba(X_test)[:, 1]

# Isotonic Regression
calibrated_isotonic = CalibratedClassifierCV(
    base_estimator=base_model,
    method='isotonic',
    cv=3 # Use 3-fold cross-validation
)
calibrated_isotonic.fit(X_train, y_train)
y_pred_proba_isotonic = calibrated_isotonic.predict_proba(X_test)[:, 1]

# 3. Visualize calibration
plt.style.use('seaborn-v0_8-whitegrid')
fig, ax = plt.subplots(figsize=(8, 8))

disp1 = CalibrationDisplay.from_predictions(y_test, y_pred_proba_uncalibrated, n_bins=10, name='Uncalibrated XGBoost', ax=ax, marker='^', color='#fa5252')
disp2 = CalibrationDisplay.from_predictions(y_test, y_pred_proba_sigmoid, n_bins=10, name='Platt Scaling', ax=ax, marker='o', color='#4dabf7')
disp3 = CalibrationDisplay.from_predictions(y_test, y_pred_proba_isotonic, n_bins=10, name='Isotonic Regression', ax=ax, marker='s', color='#69db7c')

ax.set_title('Calibration Curves (Reliability Diagram)')
ax.set_xlabel('Mean Predicted Probability')
ax.set_ylabel('Fraction of Positives')
ax.legend()
plt.grid(True)
plt.show()

Practical Notes

  • Separate Calibration Data: It's fundamental to use data for calibration that the base model hasn't seen during its training. CalibratedClassifierCV handles this via cross-validation. If you manually calibrate, ensure you use a dedicated hold-out set.
  • Choice of Method: Platt Scaling is generally preferred if you have limited calibration data or if you know the distortion is likely sigmoid. Isotonic Regression is more flexible but needs more data and can sometimes produce less smooth probability outputs. Evaluate both using appropriate metrics (like Brier score or examining reliability diagrams) on a test set.
  • Impact on Other Metrics: Calibration primarily aims to improve the reliability of probability scores (improving metrics like Brier Score or Log Loss). It might sometimes slightly decrease discriminative performance (e.g., AUC), although often the impact is minimal. The decision to calibrate depends on whether accurate probabilities are a primary requirement for your application.

In summary, probability calibration is an important post-processing technique for classification models, including gradient boosting machines. When reliable probability estimates are needed, methods like Platt Scaling or Isotonic Regression, often applied via tools like CalibratedClassifierCV, can significantly improve the trustworthiness and utility of your model's predictions.

  • Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods, John C. Platt, 1999 Advances in Neural Information Processing Systems 12, Vol. 12 (MIT Press) DOI: 10.5591/9781577353403.610 - Introduces Platt Scaling, a method to convert classifier scores into probabilities using a sigmoid function.
  • Transforming classifier scores into accurate multiclass probability estimates, Bianca Zadrozny, Charles Elkan, 2002 Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining (Association for Computing Machinery (ACM)) DOI: 10.1145/775047.775102 - Discusses using Isotonic Regression for calibrating classifier outputs into probability estimates.
  • Predicting good probabilities with supervised learning, Alexandru Niculescu-Mizil, Rich Caruana, 2005 Proceedings of the 22nd International Conference on Machine Learning (ACM) DOI: 10.1145/1102351.1102430 - A foundational paper comparing various probability calibration techniques, including Platt Scaling and Isotonic Regression, and popularizing reliability diagrams.
  • scikit-learn: sklearn.calibration.CalibratedClassifierCV, scikit-learn developers, 2024 - Official documentation for the Scikit-learn module that implements probability calibration using Platt Scaling and Isotonic Regression.

© 2026 ApX Machine Learning