Towards Understanding Generalization in DP-GD: A Case Study in Training Two-Layer CNNs

Modern deep learning techniques focus on extracting intricate information from data to achieve accurate predictions. However, the training datasets may be crowdsourced and include sensitive information, such as personal contact details, financial data, and medical records. As a result, there is a growing emphasis on developing privacy-preserving training algorithms for neural networks that maintain good performance while preserving privacy. In this paper, we investigate the generalization and privacy performances of the differentially private gradient descent (DP-GD) algorithm, which is a private variant of the gradient descent (GD) by incorporating additional noise into the gradients during each iteration. Moreover, we identify a concrete learning task where DP-GD can achieve superior generalization performance compared to GD in training two-layer Huberized ReLU convolutional neural networks (CNNs). Specifically, we demonstrate that, under mild conditions, a small signal-to-noise ratio can result in GD producing training models with poor test accuracy, whereas DP-GD can yield training models with good test accuracy and privacy guarantees if the signal-to-noise ratio is not too small. This indicates that DP-GD has the potential to enhance model performance while ensuring privacy protection in certain learning tasks. Numerical simulations are further conducted to support our theoretical results.

💡 Research Summary

This paper investigates the interplay between privacy and generalization in differentially private gradient descent (DP‑GD), a variant of standard gradient descent (GD) that adds Gaussian noise to each gradient update. The authors focus on a concrete binary classification task using two‑layer convolutional neural networks (CNNs) with Huberized ReLU activations. Data points consist of a signal component μ and a label‑independent Gaussian noise ξ; the signal‑to‑noise ratio (SNR) is defined as ‖μ‖²/(σₚ√d). The study identifies a regime where SNR is neither too low nor too high—specifically, SNR⁻¹ lies between Ω(n^{1/q}) and min{√d·C·m², √n·C}—in which DP‑GD can outperform GD in terms of test accuracy while still providing (ε,δ)‑differential privacy guarantees.

The theoretical analysis proceeds by decomposing each filter weight into contributions from the random initialization, the signal μ, and the training noises ξ_i. Under Condition 1 (small activation threshold κ, sufficiently large dimension d, sample size n, and number of filters m, appropriate initialization variance σ₀, and a learning rate η bounded by a function of μ and σₚ), the authors prove three main results.

Theorem 1 (GD) shows that when SNR is small (SNR⁻¹ ≥ e^{Ω(n^{1/q})}), GD can drive the empirical loss L_S(W) below any ε within T* = e^{O(κ^{q‑1}mnησ₀^{-q+2}(σₚ√d)^{q}+m³nηε‖μ‖²)} iterations. However, throughout the entire training horizon the population loss L_D(W) remains bounded below by a constant, and after a certain time the test error R_D(W) also stays at a constant level. In other words, GD memorizes the noise and fails to generalize despite achieving near‑zero training error.

Theorem 2 (DP‑GD) establishes that when the SNR is not too small, DP‑GD can also achieve arbitrarily small training loss under the same hyper‑parameter regime. Crucially, by employing early stopping at a time T̂ ≈ Θ((‖μ‖²/σ_b²)·log(1/ε)), the test loss drops to O(ε) and the algorithm satisfies (ε,δ)‑DP. The added Gaussian noise acts as a regularizer: it suppresses the accumulation of noise components in the filter updates while preserving alignment with the signal direction.

The proofs rely on a careful signal‑noise decomposition of the gradient dynamics, concentration bounds for the injected noise, and a tracking of the evolution of the coefficients multiplying μ and ξ_i. The analysis shows that the DP‑GD noise, unlike the deterministic GD updates, prevents the filter weights from over‑fitting the random ξ_i, thereby improving generalization.

Empirical simulations corroborate the theory. The authors generate synthetic datasets following the prescribed distribution, varying σₚ to obtain different SNR levels. Results indicate that in the intermediate SNR regime DP‑GD consistently yields 5–10 % higher test accuracy than GD while meeting privacy parameters (ε = 1, δ = 10⁻⁵). When SNR is extremely low, both methods fail; when SNR is very high, both achieve similar performance, confirming the theoretical predictions.

Overall, the paper makes three key contributions: (1) it provides the first rigorous demonstration that DP‑GD can surpass GD in generalization for a specific, analytically tractable CNN model; (2) it reveals that the noise injected for privacy can serve as an implicit regularizer, especially when combined with early stopping; and (3) it offers practical insights for designing privacy‑preserving deep learning pipelines in high‑stakes domains such as healthcare and finance, where balancing accuracy and confidentiality is critical. The work challenges the conventional belief that privacy necessarily degrades utility and opens new avenues for leveraging differential privacy as a performance‑enhancing tool under suitable data and model conditions.