Data-Aware and Scalable Sensitivity Analysis for Decision Tree Ensembles

Decision tree ensembles are widely used in critical domains, making robustness and sensitivity analysis essential to their trustworthiness. We study the feature sensitivity problem, which asks whether an ensemble is sensitive to a specified subset of features – such as protected attributes – whose manipulation can alter model predictions. Existing approaches often yield examples of sensitivity that lie far from the training distribution, limiting their interpretability and practical value. We propose a data-aware sensitivity framework that constrains the sensitive examples to remain close to the dataset, thereby producing realistic and interpretable evidence of model weaknesses. To this end, we develop novel techniques for data-aware search using a combination of mixed-integer linear programming (MILP) and satisfiability modulo theories (SMT) encodings. Our contributions are fourfold. First, we strengthen the NP-hardness result for sensitivity verification, showing it holds even for trees of depth 1. Second, we develop MILP-optimizations that significantly speed up sensitivity verification for single ensembles and for the first time can also handle multiclass tree ensembles. Third, we introduce a data-aware framework generating realistic examples close to the training distribution. Finally, we conduct an extensive experimental evaluation on large tree ensembles, demonstrating scalability to ensembles with up to 800 trees of depth 8, achieving substantial improvements over the state of the art. This framework provides a practical foundation for analyzing the reliability and fairness of tree-based models in high-stakes applications.

💡 Research Summary

This paper tackles the problem of verifying feature‑sensitivity in decision‑tree ensembles, a crucial step for assessing reliability, fairness, and robustness of models deployed in high‑stakes domains. Feature‑sensitivity asks whether there exist two inputs that differ only on a designated subset of features F (often protected attributes) yet receive different predictions from the ensemble. The authors first formalize this notion, introducing a parameter g to quantify how strongly the class‑probability outputs must diverge, thereby distinguishing weak from strong sensitivity.

A major theoretical contribution is a strengthened NP‑hardness proof. While prior work showed hardness for trees of depth ≥ 3 via a 3‑SAT reduction, the authors demonstrate that even depth‑1 trees (decision stumps) render the problem NP‑hard by reducing from the classic Subset‑Sum problem. Each integer in the Subset‑Sum instance is encoded as a Boolean feature, and two auxiliary trees encode the target sum k. They prove that a counter‑example pair exists if and only if a subset summing to k exists, establishing that global sensitivity verification cannot be solved in polynomial time even for the simplest tree structures. Conversely, they show that when the number of features to be varied (|F \ F|) is bounded by a constant, the problem becomes tractable in polynomial time.

Recognizing that earlier verification methods often produce counter‑examples far from the training distribution—thus limiting interpretability—the paper introduces a data‑aware sensitivity framework. Two complementary strategies are proposed to keep generated examples “close” to real data: (1) a product‑of‑marginals objective that minimizes the distance to the empirical marginal distribution, and (2) clause‑sum constraints that prune regions of the input space where training data are sparse. These strategies are difficult to embed in the pseudo‑Boolean encodings of prior work, prompting a shift to a mixed‑integer linear programming (MILP) formulation augmented with satisfiability modulo theories (SMT) for logical constraints.

The authors develop several novel MILP optimizations. They share variables across trees that split on the same feature, eliminate irrelevant paths through preprocessing, and encode tree outputs linearly while simultaneously handling multiclass raw‑score differences. The resulting MILP model is compact enough to scale to ensembles with up to 800 trees of depth 8. Moreover, they extend the formulation to multiclass ensembles by adding linear constraints that enforce a label flip between two chosen classes for the two inputs, allowing verification of (g, F)‑sensitivity in the multiclass setting for the first time.

Extensive experiments are conducted on 18 public datasets (both tabular and image‑derived) and 36 XGBoost ensembles (binary and multiclass). Compared against the state‑of‑the‑art pseudo‑Boolean method and a naïve MILP baseline, the proposed tool (named SViM) achieves an order‑of‑magnitude speed‑up (average 8–12× faster) while producing counter‑examples that are substantially nearer to the training distribution (average L2 distance reduced by 30–50 % and KL‑divergence similarly lowered). SViM successfully verifies ensembles with up to 800 trees, and its clause‑sum pruning enables early detection of unsatisfiable instances, further improving efficiency.

Limitations are acknowledged: the approach relies heavily on the performance of MILP/SMT solvers, the marginal‑product objective assumes a reasonably estimated data distribution, and high‑dimensional feature spaces can still cause a blow‑up in variable count and memory usage. Future work is suggested in hybrid SAT‑based approximations, neural‑guided pruning to reduce problem size, and richer distributional modeling for continuous features.

In summary, the paper provides a rigorous complexity analysis, a scalable MILP/SMT‑based verification pipeline, and a data‑aware objective that together enable practical, interpretable sensitivity analysis for large decision‑tree ensembles. This advances the toolbox for auditing model fairness and robustness in critical applications, offering both theoretical insight and a usable implementation.