PAC-Bayesian Bounds for Randomized Empirical Risk Minimizers

The aim of this paper is to generalize the PAC-Bayesian theorems proved by Catoni in the classification setting to more general problems of statistical inference. We show how to control the deviations of the risk of randomized estimators. A particular attention is paid to randomized estimators drawn in a small neighborhood of classical estimators, whose study leads to control the risk of the latter. These results allow to bound the risk of very general estimation procedures, as well as to perform model selection.

💡 Research Summary

The paper extends Catoni’s PAC‑Bayesian theorems, originally formulated for binary classification, to a broad class of statistical inference problems. The authors introduce the concept of a Randomized Empirical Risk Minimizer (RERM), which is obtained by sampling parameters from a small probabilistic neighbourhood around a classical deterministic estimator (the usual Empirical Risk Minimizer, ERM). By treating this sampling distribution as a posterior ρ and keeping an arbitrary prior π, they derive a general PAC‑Bayesian inequality that bounds the expected true risk of the randomized estimator in terms of its empirical risk, the Kullback‑Leibler divergence KL(ρ‖π), and a confidence term.

A key technical contribution is the explicit construction of ρ: instead of an arbitrary posterior, ρ is chosen to be concentrated in a ball (or ellipsoid) of radius ε around the ERM solution, often taken as a Gaussian with mean at the ERM and covariance scaled by ε. This choice yields two important benefits. First, the empirical risk under ρ is essentially the same as—or slightly lower than—that of the deterministic ERM, because random perturbations act as an implicit regularizer that mitigates over‑fitting. Second, the KL term becomes a simple function of ε and the prior, allowing a clean trade‑off between approximation accuracy and model complexity.

The authors then apply the framework to model selection. For a collection of candidate models {𝓜₁,…,𝓜_K}, each equipped with its own prior π_k and localized posterior ρ_k, they form a mixture posterior ρ = Σ_k w_k ρ_k, where the weights w_k are derived from the Bayesian updating rule. The resulting bound simultaneously controls the risk of every candidate and introduces a data‑dependent penalty that replaces traditional criteria such as AIC, BIC, or cross‑validation. The bound reads roughly as:

E_{θ∼ρ}