Sensitivity Analysis for Identification Assumptions

Identification strategies provide powerful tools like do-calculus to determine if a causal effect can be estimated from observational data, given a set of assumptions encoded in a causal graph. However, these identification assumptions, particularly the absence of unobserved confounding for a chosen adjustment set or the validity of an instrumental variable, are fundamentally untestable using the observed data alone. They rely on background knowledge, expert judgment, and hope. This raises a main question: How confident can we be in our estimated causal effects if these assumptions are slightly, or even moderately, violated? Sensitivity analysis offers a framework for answering this.

Instead of providing a single point estimate of a causal effect, sensitivity analysis examines how this estimate would change under various violations of our core identification assumptions. It allows us to quantify the strength of our conclusions. If a small, plausible violation of an assumption drastically alters the result (e.g., changes the sign of the effect or makes it statistically insignificant), our findings are considered sensitive or fragile. Conversely, if the conclusion holds even under substantial violations, we gain confidence in its reliability.

Why Identification Assumptions Need Scrutiny

Consider the standard assumption needed for identifying the Average Treatment Effect (ATE) using the backdoor criterion: conditional ignorability, which states that, conditional on a set of observed covariates $Z$, the treatment assignment $X$ is independent of the potential outcomes $Y(x)$. Formally, $Y(x) \perp X | Z$ for all $x$. This assumption implies that $Z$ blocks all backdoor paths between $X$ and $Y$, meaning $Z$ captures all common causes.

In practice, we can never be absolutely certain that our measured covariates $Z$ truly capture all common causes. There might always be an unobserved factor $U$ that affects both $X$ and $Y$, creating residual confounding even after adjusting for $Z$.

A common scenario where identification assumptions might fail. Adjusting for $Z$ does not block the backdoor path $X \leftarrow U \rightarrow Y$ because $U$ is unobserved.

Sensitivity analysis directly confronts this uncertainty. It doesn't solve the problem of unobserved confounding, but it measures its potential impact.

Quantifying Sensitivity to Unobserved Confounding

Several methods exist to assess sensitivity, primarily focusing on the potential influence of unobserved confounders.

1. Rosenbaum Sensitivity Analysis (for Matching/Stratification)

Developed initially for matched observational studies, Rosenbaum's method provides a formal way to assess how strong an unobserved confounder would need to be to undermine the study's conclusions, typically regarding the statistical significance of the treatment effect.

The core idea is to introduce a sensitivity parameter, $\Gamma \geq 1$. Imagine two units, $i$ and $j$, perfectly matched on all observed covariates $Z$. If there were no unobserved confounding, the odds of unit $i$ receiving the treatment versus unit $j$ receiving it would be equal. However, if an unobserved binary confounder $U$ exists, these odds might differ. $\Gamma$ represents the maximum odds ratio by which the treatment assignment probability could differ between two units with identical $Z$ due to differences in $U$.

• If $\Gamma = 1$, it implies no unobserved confounding influencing treatment assignment $Z$.
• If $\Gamma = 2$, it means an unobserved confounder could multiply the odds of receiving treatment by up to a factor of 2 for individuals who appear identical based on $Z$.

The analysis then calculates bounds on the p-value for the treatment effect hypothesis test for different assumed values of $\Gamma$. For example, we might find that the treatment effect remains statistically significant (e.g., p < 0.05) for $\Gamma$ up to 1.8, but becomes non-significant for $\Gamma \ge 1.8$.

Interpretation: We would conclude the finding is strong to unobserved confounders that change the odds of treatment assignment by a factor less than 1.8. To gauge if this is truly reliable, we often compare this $\Gamma$ value to the effect sizes of observed covariates. If an observed covariate strongly associated with both treatment and outcome only changes the odds by, say, 1.5, then reliability up to $\Gamma = 1.8$ seems reasonably strong.

2. Oster's Coefficient Stability Approach (for Regression)

Emily Oster proposed a method more directly applicable in regression settings, linking sensitivity to the stability of the treatment coefficient when observed controls are added. The intuition is that if the estimated treatment effect changes dramatically when we add observed covariates that explain little additional variance in the outcome, it suggests the estimate might be highly sensitive to unobserved covariates as well.

The method requires specifying two parameters:

• $\delta$: A parameter representing the degree of proportionality between how observables and unobservables are related to the treatment. $\delta = 1$ implies that the unobservables have the same relationship with the treatment as the observables do (conditional on observables). Values greater than 1 imply unobservables are more strongly related to treatment than observables. Negative values are also possible. Choosing $\delta$ is often the most challenging part, frequently set to 1 as a benchmark.
• $R_{max}^2$: The R-squared from a regression of the outcome on the treatment and both observed and unobserved covariates. This represents the total variance explainable by the model. It must be less than or equal to 1, often chosen based on domain knowledge or by comparison to R-squared values in similar studies. A common benchmark is $R_{max}^2 = 1.3 \times \tilde{R}^2$, where $\tilde{R}^2$ is the R-squared from the regression with observed controls.

Given $\delta$ and $R_{max}^2$, the method calculates the "bias-adjusted" treatment effect, $\beta^*$. We can then determine the value of $\delta$ required to make $\beta^*$ equal to zero (or cross some other threshold) for a given $R_{max}^2$.

Interpretation: If the treatment effect remains meaningfully different from zero even for plausible values of $\delta$ (like 1) and a conservatively high $R_{max}^2$, the result is considered reliable. Conversely, if even a small $\delta$ (e.g., 0.5) drives the effect to zero with a reasonable $R_{max}^2$, the finding is sensitive.

3. Visualizing Sensitivity

Graphical representations are extremely helpful. A common visualization plots the estimated treatment effect (or confidence interval bounds) against the sensitivity parameter ($\Gamma$ or $\delta$). This allows practitioners to quickly see the threshold at which their conclusions might change.

Consider a sensitivity analysis result:

Confidence interval for the estimated treatment effect as a function of a sensitivity parameter. The interval crosses zero (red dashed line) when the parameter reaches approximately 2.7.

In this plot, the blue lines represent the 95% confidence interval for the treatment effect. As the sensitivity parameter (representing the strength of potential unobserved confounding) increases along the x-axis, the confidence interval widens and shifts. The point where the lower bound crosses the zero effect line (here, around 2.7) indicates the threshold of robustness.

Sensitivity to Unobserved Confounding

While unobserved confounding is the most frequent target, sensitivity analysis can, in principle, be applied to other assumptions:

• Exclusion Restriction (IV): How much direct effect could the instrument have on the outcome, violating the exclusion restriction, before the IV estimate changes substantially?
• Parallel Trends (DiD): How different could the pre-treatment trends between the treatment and control groups be before the DiD estimate becomes unreliable?
• Graph Structure: How would the identified estimand change if certain edges were added to or removed from the assumed DAG? This is often explored by proposing alternative plausible graphs and re-running the identification process.

Practical Notes and Reporting

Implementing sensitivity analysis requires careful thought and justification for the chosen parameters ($\Gamma$, $\delta$, $R_{max}^2$).

• Benchmarking: Compare the required strength of the unobserved confounder ($\Gamma$ or $\delta$) to the strength of observed covariates. If the result is sensitive only to confounding much stronger than any observed predictor, it lends credibility.
• Software: Libraries like sensemakr (R and Python), causalToolbox (R), and functionalities within DoWhy (Python) can help implement these analyses.
• Transparency: Always report sensitivity analysis results alongside primary causal effect estimates. Clearly state the method used, the parameters chosen, the justification for those parameters, and the interpretation. This transparency is fundamental for credible causal inference from observational data.

Sensitivity analysis is not a magic bullet; it relies on scenarios. However, it transforms the discussion from an untestable binary assumption ("there is no unobserved confounding") to a quantitative assessment ("how much unobserved confounding would be needed to change our conclusion?"). This shift is essential for responsible causal inference in complex systems, providing a necessary layer of rigor and humility to our findings.