Cross-Validation with Antithetic Gaussian Randomization

We introduce a new cross-validation method based on an equicorrelated Gaussian randomization scheme. Our method is well-suited for problems where sample splitting is infeasible, either because the data violate the assumption of independent and identically distributed samples, or because there are insufficient samples to form representative train-test data pairs. In such problems, our method provides a simple, principled, and computationally efficient approach to estimating prediction error, often outperforming standard cross-validation while requiring only a small number of repetitions. Drawing inspiration from recent splitting techniques like data fission and data thinning, our method constructs train-test data pairs using Gaussian randomization. Our main contribution is the introduction of an antithetic Gaussian randomization scheme, involving a carefully designed correlation structure among the randomization variables. We show theoretically that this antithetic construction can eliminate the bias of cross-validation for a broad class of smooth prediction functions, without inflating variance. Through simulations across a range of data types and loss functions, we demonstrate that our estimator outperforms existing methods for prediction error estimation.

💡 Research Summary

The paper introduces a novel cross‑validation (CV) technique that avoids explicit data splitting, making it suitable for settings where the i.i.d. assumption is violated or the sample size is too small to form representative train‑test folds. The core idea is to add equicorrelated Gaussian noise to a sufficient statistic of the data and to construct K train‑test pairs from these noisy copies. Crucially, the noise vectors are generated with the most negative possible pairwise correlation (‑1/(K‑1)), a construction the authors call “antithetic Gaussian randomization.” This correlation structure forces the sum of the K noise vectors to be zero, so that when all folds are pooled the original data are exactly recovered, mirroring a key property of standard K‑fold CV.

Two user‑controlled parameters govern the method: α>0, which scales the magnitude of the added noise, and K≥2, the number of repetitions. α directly controls bias: as α→0 the training data become indistinguishable from the original sample, eliminating the bias that typically arises from training on noisier data. Unlike standard CV, where bias is reduced by increasing the number of folds (often at great computational cost), α can be made arbitrarily small without affecting the number of repetitions. K controls variance through averaging; even with very small α, the variance of the estimator remains stable because the antithetic correlation cancels the variability that would otherwise explode.

The authors first review the coupled bootstrap (CB) estimator, which also creates noisy train‑test pairs but uses independent Gaussian perturbations. CB is unbiased for a noise‑inflated version of the prediction error, but its variance grows like O((Kα)⁻¹) as α→0, making it impractical for low‑bias regimes. By contrast, the antithetic scheme yields an estimator CV_α whose bias vanishes as α→0 while its variance stays O(1/K) for a broad class of smooth prediction functions. The paper provides rigorous proofs of these claims, leveraging the zero‑sum constraint of the antithetic noises and smoothness assumptions on the predictor g(·).

A further theoretical contribution is the connection between CV_α and Stein’s unbiased risk estimator (SURE). The authors show that CV_α can be interpreted as SURE applied to a convolution‑smoothed version of the predictor, thereby linking the randomization approach to classical unbiased risk estimation.

Empirically, the method is evaluated on synthetic data covering isotonic regression, logistic regression, and other loss functions. In scenarios with strong dependence (time‑series, spatial data) or rare categories, where traditional sample splitting would destroy structure, the antithetic CV with K=2 and α as low as 0.01 outperforms standard 2‑fold CV, 10‑fold CV, and leave‑one‑out CV in terms of mean‑squared error, while being orders of magnitude faster. Comparisons with the coupled bootstrap and SURE confirm that the proposed method achieves lower bias without the variance blow‑up that plagues the alternatives. The authors also demonstrate that only a few repetitions (K=2 or 3) are sufficient when α is chosen appropriately, simplifying practical implementation.

The paper concludes by outlining future directions: extending the framework to non‑Gaussian or heavy‑tailed data, handling multiple loss functions simultaneously, and developing high‑dimensional theory. Overall, the antithetic Gaussian randomization provides a theoretically sound, computationally efficient, and broadly applicable alternative to traditional cross‑validation, especially in settings where data splitting is infeasible or undesirable.