A hierarchical approach for assessing the vulnerability of tree-based classification models to membership inference attack

Machine learning models can inadvertently expose confidential properties of their training data, making them vulnerable to membership inference attacks (MIA). While numerous evaluation methods exist, many require computationally expensive processes, such as training multiple shadow models. This article presents two new complementary approaches for efficiently identifying vulnerable tree-based models: an ante-hoc analysis of hyperparameter choices and a post-hoc examination of trained model structure. While these new methods cannot certify whether a model is safe from MIA, they provide practitioners with a means to significantly reduce the number of models that need to undergo expensive MIA assessment through a hierarchical filtering approach. More specifically, it is shown that the rank order of disclosure risk for different hyperparameter combinations remains consistent across datasets, enabling the development of simple, human-interpretable rules for identifying relatively high-risk models before training. While this ante-hoc analysis cannot determine absolute safety since this also depends on the specific dataset, it allows the elimination of unnecessarily risky configurations during hyperparameter tuning. Additionally, computationally inexpensive structural metrics serve as indicators of MIA vulnerability, providing a second filtering stage to identify risky models after training but before conducting expensive attacks. Empirical results show that hyperparameter-based risk prediction rules can achieve high accuracy in predicting the most at risk combinations of hyperparameters across different tree-based model types, while requiring no model training. Moreover, target model accuracy is not seen to correlate with privacy risk, suggesting opportunities to optimise model configurations for both performance and privacy.

💡 Research Summary

This paper tackles the costly problem of assessing membership inference attack (MIA) vulnerability in tree‑based classifiers by introducing a two‑stage hierarchical filtering framework. The first stage, an ante‑hoc analysis, investigates how hyper‑parameter choices alone influence privacy risk. By exhaustively training many configurations across several public datasets, the authors discover that the relative ranking of risk for different hyper‑parameter combinations is remarkably stable across data domains. Leveraging this observation, they derive simple, human‑readable decision rules (e.g., deep trees with low minimum‑sample‑per‑leaf and complex split criteria tend to be high‑risk). These rules can be applied before any model is trained, allowing practitioners to prune risky regions of the hyper‑parameter search space and thereby reduce unnecessary training effort.

The second stage, a post‑hoc analysis, examines the structure of a trained model to derive cheap proxy metrics that indicate susceptibility to MIA. The authors propose several structural measures—total node count, leaf‑node sample distribution, minimum/maximum leaf size, and a novel FDIFz metric that normalises the difference in predicted probabilities between leaves. All these metrics can be computed in negligible time once the model is available. Empirical evaluation shows that the structural metrics, especially FDIFz, correlate strongly with actual MIA success rates, achieving high precision (few false positives) but lower recall (some vulnerable models are missed). Consequently, the metrics serve as a sufficient‑but‑not‑necessary filter: models flagged as high‑risk are very likely to be vulnerable, while unflagged models may still require a full shadow‑model attack for confirmation.

Experiments span single decision trees, random forests, and gradient‑boosted trees on a variety of benchmark datasets (tabular and image‑derived). Key findings include: (1) hyper‑parameter‑based risk prediction rules achieve >80 % accuracy in identifying the most vulnerable configurations, independent of the dataset; (2) there is no statistically significant correlation between a model’s predictive accuracy and its privacy risk, suggesting that privacy‑preserving configurations can be selected without sacrificing performance; (3) structural metrics provide a fast, inexpensive way to triage models before committing to expensive shadow‑model attacks.

The paper also situates its contributions within the broader field of Statistical Disclosure Control (SDC) used in Trusted Research Environments (TREs). It maps classic SDC concepts—degrees of freedom, k‑anonymity, and class‑disclosure risk—to tree‑based models (e.g., counting the number of parameters implied by tree splits, measuring leaf‑level anonymity, and detecting overly confident class predictions). By doing so, the authors bridge regulatory practice and machine‑learning privacy assessment, offering a practical workflow for TRE staff: (i) apply ante‑hoc hyper‑parameter filters, (ii) compute post‑hoc structural metrics on trained models, and (iii) only for models that pass both filters perform full shadow‑model MIA evaluation.

In summary, the hierarchical approach dramatically reduces the computational burden of privacy risk assessment for tree‑based classifiers while providing actionable insights for both model developers and data custodians. It demonstrates that privacy risk can often be anticipated from design choices and model structure, enabling more efficient, privacy‑aware machine‑learning pipelines in regulated environments.