Can One Estimate The Unconditional Distribution of Post-Model-Selection Estimators?

We consider the problem of estimating the unconditional distribution of a post-model-selection estimator. The notion of a post-model-selection estimator here refers to the combined procedure resulting from first selecting a model (e.g., by a model selection criterion like AIC or by a hypothesis testing procedure) and then estimating the parameters in the selected model (e.g., by least-squares or maximum likelihood), all based on the same data set. We show that it is impossible to estimate the unconditional distribution with reasonable accuracy even asymptotically. In particular, we show that no estimator for this distribution can be uniformly consistent (not even locally). This follows as a corollary to (local) minimax lower bounds on the performance of estimators for the distribution; performance is here measured by the probability that the estimation error exceeds a given threshold. These lower bounds are shown to approach 1/2 or even 1 in large samples, depending on the situation considered. Similar impossibility results are also obtained for the distribution of linear functions (e.g., predictors) of the post-model-selection estimator.

💡 Research Summary

The paper tackles a fundamental question in modern statistical practice: can we consistently estimate the unconditional distribution of a post‑model‑selection estimator? A post‑model‑selection estimator is defined as the two‑stage procedure that first selects a model from a candidate set using the same data (via criteria such as AIC, BIC, cross‑validation, or hypothesis testing) and then estimates the parameters of the chosen model (by least squares, maximum likelihood, etc.). The authors ask whether, based on the observed sample, one can construct an estimator of the full sampling distribution of this combined procedure that is uniformly consistent across the entire parameter space.

To answer, they adopt a minimax decision‑theoretic framework. The loss is the probability that the estimated distribution deviates from the true distribution by more than a pre‑specified tolerance ε. By constructing two nearby parameter points θ and θ′ such that the model‑selection rule chooses different models at each point, they create a situation where the resulting post‑selection estimators have fundamentally different distributions, yet the data cannot reliably tell which scenario generated them. Under this construction they derive local minimax lower bounds on the risk. The bounds show that, for any estimator, the probability of exceeding the ε‑error threshold converges to at least ½ (and in some settings to 1) as the sample size grows. Consequently, no estimator can achieve uniform consistency—even locally—because the error probability does not vanish.

The impossibility result extends beyond the estimator itself to any linear functional of it, such as predictions ŷ = xᵀβ̂. Hence, confidence intervals or predictive distributions derived after model selection are intrinsically unreliable if they are interpreted unconditionally. The authors emphasize that the source of the difficulty is the randomness introduced by the model‑selection step; conditioning on the selected model restores identifiability, but the unconditional problem remains intractable.

Finally, the paper discusses practical implications. Since unconditional distribution estimation is impossible, analysts should either adopt conditional approaches (treating the selected model as fixed), employ Bayesian model‑averaging or posterior predictive methods, or develop new inferential tools that explicitly account for selection uncertainty. The work provides a rigorous theoretical foundation for the growing awareness that post‑selection inference must be handled with great caution, and it clarifies the limits of what can be achieved with traditional frequentist techniques.