Taxonomy of Interpretability Methods

With increasingly complex machine learning models, understanding how to interpret them becomes just as important as building them. Not all interpretation techniques are created equal. They differ in their approach, applicability, and the type of information they provide. To understand these methods better, classifying them along a few important dimensions is beneficial. This classification helps in selecting the right tool for the job based on the model you're working with and the questions you need to answer.

The primary ways to categorize interpretability methods are:

Intrinsic vs. Post-hoc
Model-Specific vs. Model-Agnostic

Let's examine each of these dimensions.

Intrinsic vs. Post-hoc Interpretability

This distinction centers on when interpretability is considered in the modeling process.

Intrinsic Interpretability

Some models are considered inherently interpretable due to their simpler structure. Their internal mechanics are relatively easy to understand and explain without needing additional tools. Examples include:

Linear Models (e.g., Linear Regression, Logistic Regression): The weights (coefficients) assigned to each feature directly indicate the feature's influence on the prediction, assuming features are scaled appropriately. A positive weight means an increase in the feature value corresponds to an increase (or higher probability) in the output, and vice-versa for negative weights. The magnitude suggests the strength of the influence.
Decision Trees: The path from the root node to a leaf node represents a series of explicit decisions based on feature values. This path provides a clear rule-based explanation for any prediction falling into that leaf.
Other rule-based systems (e.g., Naive Bayes under certain conditions): These models often rely on understandable conditional probabilities or simple rules.

The advantage of intrinsic interpretability is its directness. The explanation is the model structure. However, these simpler models might not achieve the highest predictive accuracy on datasets with complex, non-linear relationships between features. There's often a trade-off between model complexity (and potential accuracy) and built-in interpretability.

Post-hoc Interpretability

Most complex, high-performance models like deep neural networks, gradient boosting machines (GBM), or random forests often operate as 'black boxes'. Their internal decision-making processes are difficult, if not impossible, for humans to follow directly. For these models, we rely on post-hoc methods.

These techniques are applied after the model has been trained. They work by analyzing the trained model's input-output behavior without attempting to dissect its internal structure (though some methods might leverage specific aspects if available). Post-hoc methods aim to provide approximations or summaries of the model's behavior, either globally or for specific predictions.

LIME and SHAP, the main focus of this course, are prominent examples of post-hoc interpretability techniques. They provide insights into complex models that wouldn't otherwise be easily understood.

A comparison showing intrinsic interpretability deriving directly from simple models, while post-hoc methods are applied after training complex models.

Model-Specific vs. Model-Agnostic Interpretability

This dimension classifies methods based on whether they can be applied to any model or only specific types.

Model-Specific Methods

These techniques are designed for a particular class of models and often rely on leveraging the internal workings or properties unique to that model family. Examples include:

Coefficient analysis for linear models: As mentioned before, directly interpreting the learned weights.
Tree-based feature importance: Calculating importance based on how much each feature contributes to reducing impurity (like Gini impurity or entropy) across all splits in a decision tree or an ensemble of trees (like Random Forest or Gradient Boosting).
Activation maximization for neural networks: Techniques used to visualize what patterns maximally activate certain neurons or layers within a network.
TreeSHAP: While part of the SHAP framework, TreeSHAP is a highly efficient algorithm specifically designed for tree-based models, leveraging their structure.

Model-specific methods can be very efficient and provide deep insights tailored to the model type. However, their main limitation is their lack of portability. You cannot use a method designed for decision trees to interpret a Support Vector Machine (SVM) or a neural network. This also makes it difficult to compare explanations across different model types using these methods.

Model-Agnostic Methods

These methods treat the model as a black box. They work by analyzing the relationship between input variations and output changes, without needing access to the model's internal structure (like weights or decision rules). They typically require only access to the model's prediction function (predict() or predict_proba()).

Examples include:

LIME (Local Interpretable Model-agnostic Explanations): Builds simple, interpretable local surrogate models around individual predictions.
SHAP (SHapley Additive exPlanations): Uses concepts from game theory (Shapley values) to fairly distribute the prediction outcome among features. KernelSHAP is a model-agnostic implementation.
Partial Dependence Plots (PDP) and Individual Conditional Expectation (ICE) Plots: Visualize the average or individual effect of one or two features on the predicted outcome.
Permutation Feature Importance: Measures the importance of a feature by observing how much the model's performance decreases when that feature's values are randomly shuffled.

The primary advantage of model-agnostic methods is their flexibility. You can apply the same technique to interpret and compare different types of models (e.g., compare feature importance from a Random Forest vs. a Neural Network for the same task). This is valuable during model selection. The potential downside is that they might be computationally more intensive than model-specific methods (like TreeSHAP vs. KernelSHAP) and, being approximations, might sometimes miss nuances captured by methods that exploit model internals.

Model-agnostic methods can be applied to various model types, whereas model-specific methods are tailored to particular model architectures.

Understanding the Interaction

It's useful to note that these categories often intersect. For instance:

LIME and KernelSHAP are both post-hoc and model-agnostic.
Interpreting coefficients of a linear model is intrinsic and model-specific.
TreeSHAP is post-hoc but model-specific (to tree-based models).

Recognizing where different techniques fall within this taxonomy is fundamental for choosing the right interpretability approach. If you are using a simple linear model, you might rely on its intrinsic interpretability. If you have trained a complex gradient boosting model and need to explain its predictions to stakeholders, a post-hoc, model-agnostic method like LIME or SHAP (or the model-specific TreeSHAP) would be appropriate. This classification framework provides a structured way to think about the available options before exploring specific algorithms like LIME and SHAP in the following chapters.

Was this section helpful?

References

Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, Christoph Molnar, 2023 - A comprehensive online book offering a structured overview of machine learning interpretability methods, including various classification dimensions, local/global explanations, and model-specific/agnostic techniques.
"Why Should I Trust You?": Explaining the Predictions of Any Classifier, Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin, 2016 Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Association for Computing Machinery) DOI: 10.1145/2939672.2939778 - The original paper introducing LIME, a widely adopted post-hoc and model-agnostic method that generates local, interpretable explanations for individual predictions of any classifier.
A Unified Approach to Interpreting Model Predictions, Scott M Lundberg, Su-In Lee, 2017 Advances in Neural Information Processing Systems 30 (NIPS 2017) (NeurIPS) - The seminal paper presenting SHAP, a game-theoretic explanation framework that unifies various interpretability methods and provides consistent feature attributions, including model-agnostic (KernelSHAP) and model-specific (TreeSHAP) implementations.
A Survey of Explainable Artificial Intelligence (XAI): Methods, Applications, Challenges, and Opportunities, Amina Adadi, Mohammed Berrado, 2018 Applied System Innovation, Vol. 1 (MDPI) DOI: 10.3390/asi1040031 - A comprehensive survey that categorizes Explainable AI (XAI) methods based on several dimensions, including the distinction between model-agnostic/specific and pre-model/post-model approaches, offering a broad overview of the field.