Model-Agnostic Dynamic Feature Selection with Uncertainty Quantification

Dynamic feature selection (DFS) addresses budget constraints in decision-making by sequentially acquiring features for each instance, making it appealing for resource-limited scenarios. However, existing DFS methods require models specifically designed for the sequential acquisition setting, limiting compatibility with models already deployed in practice. Furthermore, they provide limited uncertainty quantification, undermining trust in high-stakes decisions. In this work, we show that DFS introduces new uncertainty sources compared to the static setting. We formalise how model adaptation to feature subsets induces epistemic uncertainty, how standard imputation strategies bias aleatoric uncertainty estimation, and why predictive confidence fails to discriminate between good and bad selection policies. We also propose a model-agnostic DFS framework compatible with pre-trained classifiers, including interpretable-by-design models, through efficient subset reparametrization strategies. Empirical evaluation on tabular and image datasets demonstrates competitive accuracy against state-of-the-art greedy and reinforcement learning-based DFS methods with both neural and rule-based classifiers. We further show that the identified uncertainty sources persist across most existing approaches, highlighting the need for uncertainty-aware DFS.

💡 Research Summary

Dynamic feature selection (DFS) is essential for budget‑constrained decision making, yet current approaches suffer from two major drawbacks: they require models specifically designed for sequential acquisition and they provide little insight into the uncertainty of their predictions. This paper first formalises the novel sources of uncertainty introduced by DFS. By training a single global predictor θ across all possible feature subsets, the authors show that each subset S incurs an epistemic risk gap Δ_S between the globally trained model and a hypothetical subset‑optimal model θ*_S. This gap quantifies the epistemic uncertainty caused by model adaptation to many environments. Moreover, standard imputation methods (mean, zero) implicitly assume a degenerate distribution for missing features, biasing the estimation of aleatoric uncertainty. The authors also demonstrate that predictive entropy is not monotonic with respect to feature acquisition; acquiring an informative but noisy test can increase entropy even while epistemic uncertainty drops.

To address these issues, the paper proposes a model‑agnostic DFS framework that works with any pre‑trained classifier, including interpretable rule‑based models. The key idea is a subset‑dependent re‑parameterisation function R_ψ(θ, S) that produces a lightweight, subset‑specific parameter set θ_S = R_ψ(θ, S). This function is trained to minimise the expected risk over subsets, thereby shrinking Δ_S and reducing the bias introduced by imputation. For neural networks, only the final layer or a low‑rank update is adapted; for rule‑based classifiers, efficient parameter tweaks are described.

The DFS policy is driven by a value function
v_q(i, x_S) = 𝔼_{p(x_i|x_S)}