Targeted Learning for Variable Importance

Variable importance is one of the most widely used measures for interpreting machine learning with significant interest from both statistics and machine learning communities. Recently, increasing attention has been directed toward uncertainty quantification in these metrics. Current approaches largely rely on one-step procedures, which, while asymptotically efficient, can present higher sensitivity and instability in finite sample settings. To address these limitations, we propose a novel method by employing the targeted learning (TL) framework, designed to enhance robustness in inference for variable importance metrics. Our approach is particularly suited for conditional permutation variable importance. We show that it (i) retains the asymptotic efficiency of traditional methods, (ii) maintains comparable computational complexity, and (iii) delivers improved accuracy, especially in finite sample contexts. We further support these findings with numerical experiments that illustrate the practical advantages of our method and validate the theoretical results.

💡 Research Summary

This paper addresses the problem of quantifying uncertainty for variable importance (VI) measures, a cornerstone of interpretable machine learning. While VI is widely used to assess the contribution of individual covariates, existing uncertainty quantification methods rely heavily on one‑step debiasing procedures that employ efficient influence functions. These methods, although asymptotically efficient, can be highly unstable in finite‑sample regimes because they depend on empirical distributions that fluctuate considerably, and they often require restrictive Donsker‑class assumptions or costly bootstrap resampling.

The authors propose a novel approach based on the Targeted Learning (TL) framework, originally developed for semiparametric causal inference. Starting from an initial estimate of the data‑generating distribution (\hat P), TL constructs a one‑dimensional parametric submodel (P_\epsilon = (1+\epsilon \hat\psi)\hat P) that moves the distribution in the direction of the efficient influence function (\hat\psi). By maximizing the empirical log‑likelihood with respect to (\epsilon), the method obtains an updated distribution that eliminates the plug‑in bias. This update is iterated until the estimated (\epsilon) becomes zero, yielding a final estimator (\hat\Psi_n = \Psi(P_k)) that is asymptotically linear, has the same efficient influence function as the one‑step estimator, and enjoys a second‑order remainder term of order (o_P(n^{-1/2})).

To make the theory practical, the authors adopt a sample‑splitting scheme: the data are divided into three disjoint subsets (I_1, I_2, I_3). The first split is used to train an initial predictive model (\hat f) (e.g., a random forest) and to estimate conditional densities (p(y|z)) and (p(x|z)). The second split drives the TL updates, providing an unbiased estimate of the influence function and the likelihood maximization step. The third split is reserved for variance estimation and confidence‑interval construction. This design relaxes Donsker conditions and allows the use of flexible machine‑learning learners that may not be asymptotically Donsker.

The methodology is illustrated on Conditional Permutation Importance (CPI), a model‑agnostic VI metric that avoids extrapolation by permuting a covariate (X) conditional on other covariates (Z). The authors derive explicit efficient influence functions for both the “permuted” loss term and the baseline loss term, and they embed these into the TL algorithm (Algorithm 1). Conditional densities are approximated via weighted empirical distributions derived from out‑of‑bag (OOB) predictions of random forests, and Monte‑Carlo integration is used to evaluate the influence‑function expectations. The TL update simply rescales the OOB weights by ((1+\epsilon \hat\psi)), making the computational overhead comparable to a single additional likelihood maximization.

Theoretical contributions include (i) proof that the TL estimator retains the asymptotic efficiency of the one‑step influence‑function estimator, (ii) demonstration that the second‑order remainder is negligible under mild regularity conditions, and (iii) validation that the variance estimator based on the third data split is consistent.

Empirical results comprise extensive simulations and two real‑world applications (a medical outcome dataset and a sociological survey). Across all settings, TL‑VI exhibits lower mean‑squared error and narrower, more accurate confidence intervals than both the one‑step debiased estimator and bootstrap‑based methods, while maintaining similar computational cost. The authors also discuss the limitation that TL requires a single representation of the joint distribution, making it unsuitable for methods like LOCO that rely on refitting models after dropping a covariate.

In summary, the paper introduces a robust, theoretically sound, and computationally efficient framework for uncertainty quantification of variable importance. By iteratively targeting the efficient influence function, the TL approach mitigates the instability of one‑step debiasing, preserves asymptotic optimality, and delivers practical gains in finite‑sample performance, especially for conditional permutation importance. Future work may extend TL‑VI to multivariate importance measures, high‑dimensional settings, and time‑to‑event outcomes.