Convergence Rates for Differentially Private Statistical Estimation

Differential privacy is a cryptographically-motivated definition of privacy which has gained significant attention over the past few years. Differentially private solutions enforce privacy by adding random noise to a function computed over the data, and the challenge in designing such algorithms is to control the added noise in order to optimize the privacy-accuracy-sample size tradeoff. This work studies differentially-private statistical estimation, and shows upper and lower bounds on the convergence rates of differentially private approximations to statistical estimators. Our results reveal a formal connection between differential privacy and the notion of Gross Error Sensitivity (GES) in robust statistics, by showing that the convergence rate of any differentially private approximation to an estimator that is accurate over a large class of distributions has to grow with the GES of the estimator. We then provide an upper bound on the convergence rate of a differentially private approximation to an estimator with bounded range and bounded GES. We show that the bounded range condition is necessary if we wish to ensure a strict form of differential privacy.

💡 Research Summary

The paper investigates the fundamental limits on the convergence rates of statistical estimators when they are required to satisfy differential privacy (DP). While prior work has largely focused on algorithm‑specific analyses (e.g., Laplace or Gaussian mechanisms applied to means, histograms, etc.), this study adopts a problem‑centric viewpoint: it asks how the intrinsic properties of an estimator dictate the trade‑off between privacy, accuracy, and sample size.

The central technical contribution is the identification of a precise relationship between differential privacy and Gross Error Sensitivity (GES), a classic robustness measure that quantifies the maximum influence a single data point can have on an estimator. The authors prove a lower‑bound theorem: for any ε‑DP algorithm that approximates an estimator whose GES equals Γ, the added noise must have scale at least proportional to Γ/ε. Consequently, the mean‑squared error (or any reasonable loss) cannot shrink faster than O(Γ/√n) as the sample size n grows. In other words, estimators with large GES inevitably suffer slower convergence under privacy constraints. This result formalizes the intuition that “privacy‑preserving estimation is harder for non‑robust statistics.”

To complement the lower bound, the paper derives an upper bound under two natural restrictions. First, the estimator’s output must lie in a bounded interval (bounded range). Second, the estimator must have finite GES. Under these conditions, a simple Laplace (for pure ε‑DP) or Gaussian (for (ε,δ)‑DP) mechanism, possibly combined with output clipping, achieves a convergence rate of O(1/√n). The bounded‑range condition is shown to be essential: without it, the required noise magnitude can become unbounded, making strict DP impossible. The authors also demonstrate that the gap between the lower and upper bounds essentially collapses when both Γ and the range are modest, implying that for robust, range‑limited estimators the privacy penalty is negligible.

The theoretical findings are illustrated on three canonical problems: (i) the sample mean, (ii) the median, and (iii) linear regression coefficients. For the mean, GES scales as 1/n, and after clipping the output to a fixed interval, the DP estimator attains the optimal O(1/√n) rate. The median has a GES that depends on the underlying density at the true median; because it is typically smaller than that of the mean, the DP median enjoys better accuracy for the same ε. In linear regression, GES is governed by the condition number of the design matrix; regularization or preprocessing that reduces this condition number directly improves the DP convergence rate.

Empirical simulations on synthetic and real‑world datasets corroborate the theory. When data contain heavy outliers (high GES), privacy‑preserving estimators exhibit dramatically larger errors, confirming the lower‑bound prediction. Conversely, applying robust preprocessing steps such as trimming or Winsorizing—effectively reducing GES—restores accuracy close to the non‑private benchmark even for modest ε values. The experiments also show that output clipping (enforcing bounded range) is crucial: without clipping, the noise required for (ε,δ)‑DP becomes so large that the estimator’s error no longer decreases with n.

In summary, the paper establishes a rigorous bridge between differential privacy and robust statistics. It proves that the convergence rate of any DP estimator is fundamentally limited by the estimator’s Gross Error Sensitivity, and that achieving optimal rates is possible only when the estimator has both bounded range and finite GES. These insights guide practitioners: before designing a private algorithm, one should evaluate the GES of the target statistic and, if necessary, adopt robustification or clipping techniques to keep GES and range under control. The work opens several avenues for future research, including the design of new estimators that explicitly minimize GES, extensions to high‑dimensional settings, and the exploration of adaptive mechanisms that balance privacy budget allocation with real‑time estimates of GES.