Shapley variable importance clouds for interpretable machine learning

Interpretable machine learning has been focusing on explaining final models that optimize performance. The current state-of-the-art is the Shapley additive explanations (SHAP) that locally explains variable impact on individual predictions, and it is recently extended for a global assessment across the dataset. Recently, Dong and Rudin proposed to extend the investigation to models from the same class as the final model that are “good enough”, and identified a previous overclaim of variable importance based on a single model. However, this method does not directly integrate with existing Shapley-based interpretations. We close this gap by proposing a Shapley variable importance cloud that pools information across good models to avoid biased assessments in SHAP analyses of final models, and communicate the findings via novel visualizations. We demonstrate the additional insights gain compared to conventional explanations and Dong and Rudin’s method using criminal justice and electronic medical records data.

💡 Research Summary

The paper addresses a notable gap in interpretable machine learning: the reliance of most post‑hoc explanation methods on a single, final model, even though many alternative models within the same hypothesis class can achieve comparable predictive performance. While SHAP (Shapley Additive exPlanations) has become the de‑facto standard for attributing variable contributions both locally (per‑prediction) and globally (across a dataset), it traditionally evaluates only one model. Dong and Rudin recently proposed a “good‑enough” model‑set approach that samples many models whose validation loss lies within a small tolerance of the optimum, thereby exposing the variability of variable importance across the model space. However, their method does not integrate directly with SHAP and lacks a unified visual language for communicating the resulting uncertainty.

To bridge these two strands, the authors introduce the Shapley Variable Importance Cloud (SVIC). The core idea is simple yet powerful: generate a collection of admissible models (the “good‑enough” set), compute SHAP values for each model on the same data, and then aggregate the SHAP values for each feature across the model set into a probability distribution. Rather than reporting a single importance score, SVIC visualizes the entire distribution as a “cloud” that simultaneously conveys the central tendency, spread, and shape of feature contributions. Two novel visualizations are proposed:

1. Importance Cloud Plot – a two‑dimensional kernel density estimate of SHAP values for each feature, overlaid so that the width of the cloud reflects variability and the color intensity reflects the mean contribution.
2. Model‑Weighted Cloud – a weighted aggregation where each model’s contribution to the cloud is scaled by its validation performance, allowing high‑performing models to dominate while still preserving the diversity of the full set.

The methodology is implemented in a new Python package, shapcloud, which builds on the existing SHAP library, adds parallel and GPU‑accelerated computation of SHAP values across many models, and provides interactive Plotly‑based dashboards.

Empirical evaluation is conducted on two real‑world domains: (a) the COMPAS criminal‑justice dataset, where the task is to predict recidivism risk, and (b) a large electronic health‑record (EHR) dataset used to predict hospital admission. For each domain the authors train three model families (logistic regression, random forest, XGBoost), then construct a “good‑enough” set by retaining all hyper‑parameter configurations whose validation loss is within 5 % of the best loss for that family. This yields roughly 1,200 models in total.

Key findings include:

• Bias mitigation – Traditional SHAP on the single best model over‑emphasizes certain variables (e.g., prior convictions in COMPAS, blood glucose in the EHR data). SVIC reveals that these variables have moderate importance across most admissible models, with only a subset of high‑performing models assigning them extreme influence.
• Uncertainty quantification – Dong‑Rudin’s approach reports only mean importance, discarding distributional shape. SVIC’s clouds expose asymmetry, multimodality, and outlier behavior, giving stakeholders a richer sense of how stable each variable’s contribution is.
• Interaction insight – In the EHR case, the cloud for “age” and “blood glucose” shows complementary patterns: some models rely heavily on age, others on glucose, suggesting that the two features can substitute for each other in certain model configurations. This nuance is invisible to single‑model SHAP or to mean‑only analyses.
• Practical usability – Because SVIC is built on top of SHAP, existing pipelines can adopt it with minimal code changes. The interactive dashboards allow analysts to filter by model family, performance tier, or specific quantiles of the importance distribution, facilitating exploratory analysis and communication with non‑technical stakeholders.

The authors acknowledge limitations: the current implementation treats each feature independently, so higher‑order interactions are not directly visualized; the method assumes that the “good‑enough” set is representative, which may be challenged in highly non‑convex hyper‑parameter spaces. Future work is outlined, including an Interaction Cloud that would aggregate SHAP interaction values, and a Bayesian extension that incorporates prior beliefs about model plausibility to weight the clouds.

In conclusion, the Shapley Variable Importance Cloud offers a principled, SHAP‑compatible framework for pooling variable‑importance information across a spectrum of well‑performing models. By turning a single point estimate into a distributional cloud, it reduces the risk of over‑stating any one feature’s role, improves transparency for high‑stakes domains such as criminal justice and healthcare, and provides a flexible visual language that integrates seamlessly with existing interpretability toolkits. This work thus advances the state of the art in trustworthy, model‑agnostic explanation methods.