Adapting Selection Bias Correction Methods

Selection bias presents a distinct challenge compared to confounding bias, although both stem from issues related to unobserved factors. While confounding involves variables affecting both treatment and outcome, selection bias arises when the probability of being included in the analysis sample ($S=1$) is itself dependent on the treatment ($T$), the outcome ($Y$), or variables associated with both, often involving unobserved factors $U$. This means the analyzed sample is systematically different from the target population for which we want to draw causal conclusions. Standard estimators applied to such selected samples, even after adjusting for observed covariates $X$, can yield biased estimates of causal effects like the Average Treatment Effect (ATE) or Conditional Average Treatment Effect (CATE).

Consider estimating the effectiveness of a new online learning platform ($T$) on student test scores ($Y$). If only students who feel they benefited significantly (implying higher potential $Y$) choose to report their scores, the sample is selected based on factors related to the outcome. Similarly, estimating the return on investment for a startup accelerator program ($T$) based on the revenue ($Y$) of participating companies might suffer selection bias if only successful companies ($S=1$ depends on high $Y$) agree to share data.

The Classical Approach: Heckman Correction

The foundational work on addressing selection bias comes from econometrics, specifically the Heckman two-step correction procedure (often called Heckit). The core idea is to explicitly model the selection process alongside the outcome process.

Assume there's an underlying latent variable $S^*$ determining selection: $S^*_i = Z_i \gamma + \nu_i$ An individual $i$ is selected into the sample ($S_i=1$) if $S^*_i > 0$. Here, $Z_i$ represents variables influencing selection, $\gamma$ are coefficients, and $\nu_i$ is an error term.

The outcome model for the entire population (selected or not) is: $Y_i = X_i \beta + T_i \alpha + \epsilon_i$ Where $X_i$ are covariates, $T_i$ is the treatment, $\beta$ and $\alpha$ are parameters (with $\alpha$ being the causal effect of interest), and $\epsilon_i$ is the error term.

Selection bias occurs if the error terms of the selection and outcome models, $\nu_i$ and $\epsilon_i$, are correlated, typically because both are influenced by common unobserved factors $U$. If we naively estimate the outcome model using only the selected sample ($S_i=1$), the expected value of the error term is non-zero: $E[\epsilon_i | S_i=1] = E[\epsilon_i | Z_i \gamma + \nu_i > 0] \neq 0$

Under the assumption that $\nu_i$ and $\epsilon_i$ follow a bivariate normal distribution with correlation $\rho$, Heckman showed that this conditional expectation can be expressed as: $E[\epsilon_i | S_i=1] = \rho \sigma_\epsilon \lambda(Z_i \gamma)$ where $\sigma_\epsilon$ is the standard deviation of $\epsilon_i$, and $\lambda(\cdot)$ is the Inverse Mills Ratio (IMR): $\lambda(c) = \frac{\phi(c)}{\Phi(c)}$ Here, $\phi$ is the standard normal probability density function (PDF), and $\Phi$ is the standard normal cumulative distribution function (CDF).

The Heckman two-step procedure involves:

1. Selection Model: Estimate the selection equation using a Probit model (which assumes normally distributed errors $\nu_i$) on the full sample (both selected and non-selected units, if available) to get estimates $\hat{\gamma}$. Calculate the predicted IMR, $\hat{\lambda}_i = \lambda(Z_i \hat{\gamma})$, for the selected units ($S_i=1$).
2. Outcome Model: Estimate the outcome equation using Ordinary Least Squares (OLS) on the selected sample only, including the calculated IMR ($\hat{\lambda}_i$) as an additional regressor: $Y_i = X_i \beta + T_i \alpha + \theta \hat{\lambda}_i + \text{error}_i$ The coefficient $\hat{\alpha}$ from this regression is the bias-corrected estimate of the treatment effect. The significance of $\hat{\theta}$ serves as a test for selection bias ($\theta = \rho \sigma_\epsilon$).

An important requirement for identification in the classical Heckman model is the exclusion restriction: there must be at least one variable in $Z$ that influences selection ($S$) but does not directly influence the outcome $Y$, except possibly through its effect on selection itself (i.e., it's not included in $X$). This variable acts like an instrument for the selection process.

Adapting Correction Methods for Machine Learning

While elegant, the classical Heckman procedure relies on strong assumptions that are often difficult to justify in complex machine learning contexts:

• Distributional Assumptions: Joint normality of error terms ($\nu_i, \epsilon_i$).
• Linearity: Linear relationships in both selection and outcome models.
• Exclusion Restriction: Finding a valid variable that affects selection but not the outcome can be very challenging.

Modern ML applications often involve high-dimensional data, non-linear relationships, and complex interactions, demanding more flexible approaches. Adapting selection bias correction involves replacing the rigid parametric models with ML algorithms while retaining the core idea of modeling selection and correcting the outcome prediction.

Modeling Selection with Classifiers

Instead of a Probit model, we can use flexible binary classifiers to model the selection probability $P(S=1 | Z)$. Suitable models include:

• Logistic Regression (less flexible but interpretable)
• Gradient Boosting Machines (e.g., LightGBM, XGBoost)
• Random Forests
• Neural Networks

The input features $Z$ would include all observed variables believed to influence sample inclusion. The output of this model gives us an estimated probability of selection, $\hat{p}(Z_i) = \hat{P}(S=1 | Z_i)$.

Correcting the Outcome Model

With a selection model based on ML, we can adapt the correction step:

1. Using Inverse Probability of Selection Weighting (IPSW): If selection depends only on observed pre-treatment covariates $Z$ (which might include $X$ and potentially treatment $T$, if selection happens after treatment assignment), one might consider weighting the selected sample by the inverse of their estimated selection probability: $w_i = 1 / \hat{p}(Z_i)$. The outcome model is then trained on the selected sample using these weights. However, this approach requires the selection mechanism to be independent of the outcome $Y$ given $Z$, i.e., "selection on observables". This is a strong assumption often violated in practice (e.g., if people self-select into the study based on their perceived outcome). It also requires positivity ($P(S=1|Z) > 0$ for all relevant $Z$).

2. Control Function Approach (Generalized IMR): This is closer in spirit to the original Heckman idea. We need a way to capture the conditional expectation of the outcome error given selection, $E[\epsilon_i | S_i=1, Z_i]$. Without the normality assumption, the IMR derived earlier doesn't hold directly. However, we can use functions of the estimated selection probability $\hat{p}(Z_i)$ or the classifier's raw score as a "control function" included as an additional feature in the outcome model.

   • Train a classifier for $P(S=1 | Z)$ on the full sample (or a relevant sample allowing estimation of selection probabilities).
   • For the selected units ($S_i=1$), compute a control function term, $g(\hat{p}(Z_i))$. This could be $\hat{p}(Z_i)$ itself, $\log(\hat{p}(Z_i))$, or other transformations. The optimal form is generally unknown without strong assumptions.
   • Train a flexible ML model (e.g., Random Forest, Gradient Boosting, Neural Network) for the outcome $Y$ using the selected sample ($S_i=1$). Include the observed covariates $X_i$, treatment $T_i$, and the computed control function term $g(\hat{p}(Z_i))$ as input features. Y \sim \text{ML_Model}(X, T, g(\hat{p}(Z))) The causal effect can then be estimated by comparing predictions for treated ($T=1$) versus control ($T=0$) units, potentially averaging over the covariate distribution of interest.

Challenges and Approaches

• Exclusion Restriction: The need for a variable affecting selection but not the outcome remains a significant challenge, even with ML models. Without it, identification often relies heavily on the functional form assumptions of the control function or specific distributional properties, making the results less credible. If such a variable exists, it should be included in $Z$ for the selection model but excluded from $X$ in the outcome model.
• Data Requirements: Estimating the selection model often requires data on both selected ($S=1$) and non-selected ($S=0$) units, or at least knowledge of the features $Z$ for a representative sample of the target population. This data is frequently unavailable.
• Model Specification: Choosing the right ML models for both selection and outcome, and determining the appropriate control function transformation $g(\cdot)$, requires careful consideration and validation. There's a risk of overfitting or model misspecification bias.
• Sensitivity Analysis: Given the strong, often untestable assumptions (especially regarding the selection mechanism and the role of unobservables), sensitivity analysis is critical. How much do the estimated causal effects change if we alter the assumptions about the correlation between selection and outcome errors, or the choice of control function?

Implementation Sketch

Let's outline the steps using Python library calls for the control function approach:

# Assume:
# df_full contains features Z for selection modeling (all units)
# df_selected contains features X, treatment T, outcome Y, and Z (selected units only)
# 'selected' column in df_full indicates S=1 or S=0

from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.model_selection import train_test_split
import numpy as np
import pandas as pd

# 1. Train Selection Model
# Features for selection model
Z_features = ['z1', 'z2', 'instrument'] # 'instrument' ideally satisfies exclusion restriction
# Target for selection model
selection_target = 'selected'

selector_model = RandomForestClassifier(n_estimators=100, random_state=42)
selector_model.fit(df_full[Z_features], df_full[selection_target])

# 2. Compute Control Function for Selected Sample
# Predict selection probability for the selected units
selection_probs_selected = selector_model.predict_proba(df_selected[Z_features])[:, 1]

# Define a control function (e.g., probability itself, or logit)
# Avoid division by zero or log(0)
epsilon = 1e-6
control_function = np.log(selection_probs_selected / (1 - selection_probs_selected + epsilon))
# Alternative: control_function = selection_probs_selected

df_selected['control_func'] = control_function

# 3. Train Outcome Model with Control Function
# Features for outcome model (excluding the instrument from Z if used)
X_features = ['x1', 'x2']
T_feature = 'treatment'
outcome_target = 'outcome'

# Combine features for the outcome model
outcome_features = X_features + [T_feature, 'control_func']

outcome_model = RandomForestRegressor(n_estimators=100, random_state=123)
outcome_model.fit(df_selected[outcome_features], df_selected[outcome_target])

# 4. Estimate Causal Effect (e.g., ATE)
# Create counterfactual dataframes for prediction
df_cf_treated = df_selected[outcome_features].copy()
df_cf_treated[T_feature] = 1

df_cf_control = df_selected[outcome_features].copy()
df_cf_control[T_feature] = 0

# Predict potential outcomes
y_hat_treated = outcome_model.predict(df_cf_treated)
y_hat_control = outcome_model.predict(df_cf_control)

# Estimate ATE on the selected sample
ate_estimate = np.mean(y_hat_treated - y_hat_control)

print(f"Estimated ATE (adjusted for selection bias): {ate_estimate:.4f}")

"> This code sketch illustrates the workflow. Implementation requires careful feature engineering, hyperparameter tuning, cross-validation, and standard error estimation (often requiring bootstrapping due to the two-stage nature)."

Summary and Alternatives

Adapting selection bias correction methods using ML offers more flexibility than classical approaches but inherits the fundamental challenge of relying on strong, often untestable, assumptions about the selection process and the role of unobserved variables. The requirement for an exclusion restriction is particularly difficult to meet convincingly in many ML settings.

Therefore, these methods should be applied cautiously, always accompanied by rigorous sensitivity analyses. When possible, alternative strategies discussed in this chapter, such as Instrumental Variables (if a valid instrument exists), Regression Discontinuity Designs (if a sharp assignment threshold exists), Difference-in-Differences (if panel data and parallel trends assumptions hold), or Proximal Causal Inference (if suitable proxy variables are available), might offer more credible routes to identification by leveraging different types of data structures or assumptions that avoid explicitly modeling the selection mechanism based solely on correlations. The choice of method depends heavily on the specific problem context, data availability, and the plausibility of the underlying assumptions.