Debiased Bayesian Inference for High-dimensional Regression Models

There has been significant progress in Bayesian inference based on sparsity-inducing (e.g., spike-and-slab and horseshoe-type) priors for high-dimensional regression models. The resulting posteriors, however, in general do not possess desirable frequentist properties, and the credible sets thus cannot serve as valid confidence sets even asymptotically. We introduce a novel debiasing approach that corrects the bias for the entire Bayesian posterior distribution. We establish a new Bernstein-von Mises theorem that guarantees the frequentist validity of the debiased posterior. We demonstrate the practical performance of our proposal through Monte Carlo simulations and two empirical applications in economics.

💡 Research Summary

This paper addresses a critical limitation in Bayesian inference for high-dimensional linear regression models. While sparsity-inducing priors like spike-and-slab and horseshoe have led to significant advances in estimation and model selection, the resulting posterior distributions and their credible intervals generally lack desirable frequentist properties. They often fail to provide valid confidence sets with correct coverage even in large samples, limiting their reliability for rigorous statistical inference.

The authors propose a novel solution: a debiasing procedure applied to the entire Bayesian posterior distribution, not just a point estimator. The method begins by obtaining an initial posterior distribution for the regression coefficients β using any preferred sparsity-inducing prior, potentially via computationally efficient approximate methods like variational Bayes. Then, for each draw from this posterior, a debiased version is computed using a correction term inspired by frequentist debiased estimators. A key innovation is the use of Bayesian bootstrap weights within this correction, which preserves posterior uncertainty and provides a natural Bayesian interpretation of the procedure. This post-processing step is computationally lightweight and can be seamlessly integrated with existing Bayesian computation workflows.

The core theoretical contribution is the establishment of a new Bernstein-von Mises theorem for the debiased posterior. Under conditions that allow the number of covariates (p) to potentially exceed the sample size (n) and grow exponentially with n, the theorem proves that the debiased posterior distribution for a parameter of interest (or a growing set of parameters) is asymptotically normal. Crucially, its asymptotic variance matches that of efficient frequentist estimators. This guarantees that credible intervals constructed from the debiased posterior achieve correct frequentist coverage asymptotically, effectively bridging the gap between Bayesian and frequentist inference in high-dimensional settings. The theory shows that the method can match the performance of state-of-the-art frequentist methods like double machine learning.

The paper demonstrates the practical utility of the approach through Monte Carlo simulations and two empirical applications in economics. The simulations show that the debiased Bayesian procedure substantially improves coverage rates compared to standard Bayesian methods based on raw sparsity-inducing posteriors. It remains competitive with, and in some cases (e.g., when regression coefficients are large) superior to, frequentist debiasing techniques. The empirical examples illustrate how the method can deliver reliable inference on a specific coefficient of interest in the presence of high-dimensional controls, a common scenario in applied econometrics.

In summary, this work provides a principled, general, and computationally feasible framework for obtaining valid frequentist inference from Bayesian posteriors in high-dimensional regression. It enhances the practical value of Bayesian methods by ensuring their inferential conclusions are robust from a repeated-sampling perspective, making them a more compelling tool for applied researchers.