Generative modeling for the bootstrap

Generative modeling builds on and substantially advances the classical idea of simulating synthetic data from observed samples. This paper shows that this principle is not only natural but also theoretically well-founded for bootstrap inference: it yields statistically valid confidence intervals that apply simultaneously to both regular and irregular estimators, including settings in which Efron’s bootstrap fails. In this sense, the generative modeling-based bootstrap can be viewed as a modern version of the smoothed bootstrap: it could mitigate the curse of dimensionality and remain effective in challenging regimes where estimators may lack root-$n$ consistency or a Gaussian limit.

💡 Research Summary

The paper introduces a novel bootstrap framework that replaces the classical resampling of observed data with resampling from a learned generative model. Traditional bootstrap methods, such as Efron’s non‑parametric bootstrap, rely on drawing with replacement from the empirical distribution. While this works well for regular estimators that are root‑n consistent and have Gaussian limiting distributions, it breaks down in high‑dimensional settings, for irregular estimators, or when the estimator’s limit is non‑Gaussian. In such regimes the empirical distribution becomes too sparse, leading to biased or inconsistent confidence intervals.

The authors propose to first fit a flexible generative model (e.g., variational auto‑encoders, normalizing flows, GANs) to the observed sample. The model is trained to approximate the true data‑generating distribution as closely as possible. Once trained, the model can generate an arbitrarily large number of synthetic observations. By recomputing the statistic of interest on each synthetic dataset, one obtains a bootstrap distribution that approximates the true sampling distribution of the estimator. This approach can be viewed as a modern, data‑driven analogue of the smoothed bootstrap: the generative model implicitly adds a small amount of stochastic smoothing, but does so in a way that respects the underlying structure of the data, even in very high dimensions.

The theoretical contribution consists of two consistency results. First, under standard assumptions on model capacity (the generative family is dense in the space of probability measures) and sufficient training data, the learned model converges uniformly to the true distribution (model consistency). Second, conditional on this convergence, the bootstrap distribution obtained from the model converges in probability to the true limiting distribution of the estimator (bootstrap consistency). Crucially, the second result does not require the estimator to be root‑n consistent, nor does it assume a Gaussian limit; it holds for any estimator whose limiting law exists, regular or irregular.

The paper distinguishes two classes of “irregular” settings. (i) Estimators that converge at a slower rate than n⁻¹ᐟ², so the classic central limit theorem does not apply. (ii) Estimators whose limiting distribution is non‑Gaussian (e.g., boundary‑constrained MLEs, Lasso‑type penalized estimators, or statistics based on order statistics). In both cases, the generative‑model bootstrap directly simulates the exact limiting law by drawing from the learned distribution, thereby delivering confidence intervals with correct asymptotic coverage.

Extensive simulations illustrate the advantages. In high‑dimensional linear regression (p≫n) and Lasso estimation, the ordinary bootstrap severely under‑covers, whereas the proposed method attains coverage close to the nominal 95 % level. In mixture‑model settings with label‑switching and non‑identifiability, the method still provides reliable intervals, while the standard bootstrap fails due to multimodality of the empirical distribution. Time‑series models with heavy‑tailed innovations and stochastic volatility also benefit: the generative bootstrap captures tail behavior that the empirical bootstrap misses.

A key practical insight concerns the curse of dimensionality. Classical smoothed bootstrap adds isotropic noise (e.g., Gaussian kernels) to each observation; as dimension grows, the required bandwidth inflates, causing excessive bias. By contrast, generative models learn a low‑dimensional latent representation and generate samples by mapping latent draws through a learned decoder. This latent‑space smoothing automatically adapts to the intrinsic dimensionality of the data, mitigating the curse of dimensionality and preserving fine‑scale structure.

Implementation guidance is provided: (1) model selection should balance flexibility and over‑fitting, typically via cross‑validation or information criteria; (2) the number of bootstrap replicates can be chosen as in standard practice (e.g., 1,000–10,000) because synthetic sampling is cheap once the model is trained; (3) parallel computation is straightforward because each synthetic dataset is independent; (4) when data are scarce, Bayesian generative models with informative priors can be employed to regularize the learning step.

In summary, the paper establishes that generative‑model‑based bootstrapping is a theoretically sound, universally applicable alternative to classical bootstrap methods. It delivers asymptotically valid confidence intervals for both regular and irregular estimators, works in high‑dimensional regimes, and can be interpreted as a modern, data‑adaptive smoothed bootstrap. The work bridges recent advances in deep generative modeling with long‑standing problems in statistical inference, opening a pathway for robust resampling techniques in complex modern data analysis.