Survival Objective Functions (Cox PH)

To train gradient boosting models for time-to-event data, specific objective functions are required. These functions often adapt the principles of the Cox Proportional Hazards model.

The Cox Proportional Hazards Model Foundation

Before integrating it into boosting, let's briefly recall the Cox Proportional Hazards (Cox PH) model. It's a semi-parametric model widely used in survival analysis. Its core idea is to model the hazard rate $h(t | X)$ for an individual with covariates $X$ at time $t$. The hazard rate represents the instantaneous risk of experiencing the event at time $t$, given survival up to that time.

The Cox model assumes proportional hazards. This means the effect of covariates is multiplicative and constant over time relative to a baseline hazard function $h_0(t)$:

$$h(t | X) = h_0(t) \exp(X \beta)$$

Here:

• $h_0(t)$ is the baseline hazard function, representing the hazard for an individual with all covariates equal to zero. It's unspecified in the standard Cox model.
• $X$ is the vector of covariates for an individual.
• $\beta$ is the vector of coefficients representing the effect size of each covariate.
• $\exp(X \beta)$ is the hazard ratio. It quantifies how the covariates scale the baseline hazard. For example, if $\exp(\beta_k) = 2$, a one-unit increase in covariate $X_k$ doubles the hazard rate at any given time $t$, holding other covariates constant.

A significant aspect of the Cox model is that it allows estimating the coefficients $\beta$ without estimating the baseline hazard $h_0(t)$. This is achieved by maximizing a partial likelihood function, which compares the risk of the individual experiencing an event at a specific time to the risks of all individuals still at risk at that time.

Adapting Cox PH for Gradient Boosting

Gradient boosting models learn a function $F(X)$ that predicts an outcome. In the context of survival analysis using a Cox-like approach, the boosting model $F(X)$ takes the role of the linear predictor $X \beta$ in the traditional Cox model. The hazard function is then modeled as:

$$h(t | X) = h_0(t) \exp(F(X))$$

The goal is to learn the function $F(X)$ using the boosting algorithm. To do this, we need an objective function based on the Cox partial likelihood.

Let's consider a dataset with $N$ individuals. For each individual $i$, we have:

• $T_i$: The observed time (either time of event or time of censoring).
• $\delta_i$: An event indicator (1 if the event was observed at $T_i$, 0 if censored).
• $X_i$: The vector of covariates.

The partial likelihood function for the Cox model is constructed by considering each time $t_i$ where an event occurs ($\delta_i = 1$). For each such event time, the likelihood compares the hazard of the individual $i$ who experienced the event to the sum of hazards of all individuals still at risk just before time $t_i$. The set of individuals at risk at time $t_i$, denoted $R_i$, includes all individuals $j$ whose observed time $T_j$ is greater than or equal to $t_i$ (i.e., they haven't experienced the event and haven't been censored before $t_i$).

The partial likelihood is:

$$L = \prod_{i: \delta_i=1} \frac{\exp(F(X_i))}{\sum_{j \in R_i} \exp(F(X_j))}$$

Gradient boosting aims to minimize a loss function, which is typically the negative log-likelihood. Taking the negative logarithm of the partial likelihood gives us the objective function to minimize:

$$Obj = - \log(L) = - \sum_{i: \delta_i=1} \left[ F(X_i) - \log \left( \sum_{j \in R_i} \exp(F(X_j)) \right) \right]$$

This expression serves as the loss function for training a gradient boosting model for survival analysis under the Cox proportional hazards framework.

Gradients and Hessians for Boosting

To optimize this objective using gradient boosting, we need its first and second derivatives (gradient and Hessian) with respect to the model's output $F(X_k)$ for each observation $k$. These derivatives guide the training of the weak learners (typically decision trees) at each boosting iteration.

The calculation involves differentiating the negative log partial likelihood term. While the full derivation can be intricate, the resulting gradient $g_k$ and Hessian $h_k$ for an observation $k$ depend on whether individual $k$ experienced an event and its relationship to the risk sets $R_i$ of individuals $i$ who did experience events. Specifically, the derivatives for $F(X_k)$ involve sums over the terms $\frac{\exp(F(X_k))}{\sum_{j \in R_i} \exp(F(X_j))}$ for all event times $t_i$ where individual $k$ was part of the risk set $R_i$.

Libraries like XGBoost and LightGBM implement these calculations internally when the Cox objective is selected. The base learners are then trained to predict the negative gradient, and the model $F(X)$ is updated iteratively.

Implementation in Boosting Libraries

Major gradient boosting libraries provide built-in support for survival analysis using the Cox objective.

• XGBoost: Use the objective parameter objective='survival:cox'. The input labels should typically be positive values for event times and negative values for censored times (e.g., time t for event, -t for censoring). Check the XGBoost documentation for the precise format expected by the version you are using.
• LightGBM: Use objective='coxph'. LightGBM usually expects two separate columns for the label: one for the time and one for the event indicator (0 for censoring, 1 for event).
• CatBoost: CatBoost also supports Cox regression via the loss_function='Cox' parameter. Similar to LightGBM, it typically expects time and event indicator columns.

The output of the trained boosting model, predict(X), provides the log relative risk $F(X)$. A higher value indicates a higher predicted risk of experiencing the event, relative to the baseline hazard. These values can be used to rank individuals by risk or potentially to estimate survival probabilities if combined with an estimate of the baseline hazard (though estimating $h_0(t)$ is often done separately after fitting the boosting model).

Important Notes

When using boosting with Cox PH objectives, remember:

• Proportional Hazards Assumption: Gradient boosting does not eliminate the need to check the proportional hazards assumption. While the flexibility of trees can capture complex interactions that might approximate non-proportionality to some extent, formally testing this assumption (e.g., using Schoenfeld residuals) is still advisable.
• Handling Tied Event Times: When multiple individuals experience an event at the exact same time, the standard partial likelihood needs modification. Common approximations like Breslow's or Efron's methods are typically used by the libraries to handle ties efficiently. Be aware of which method your chosen library uses by default.
• Evaluation: Standard metrics like accuracy or AUC are not suitable for survival analysis. Use metrics designed for censored data, such as the Concordance Index (C-index) or time-dependent AUC, to evaluate model performance. Many libraries allow specifying these metrics during training for monitoring or early stopping.

By implementing Cox PH objective functions, gradient boosting frameworks become powerful tools for analyzing time-to-event data, extending their applicability to domains like medical research, predictive maintenance, and customer churn analysis where censoring is common.