Learning in a Large Function Space: Privacy-Preserving Mechanisms for SVM Learning

Several recent studies in privacy-preserving learning have considered the trade-off between utility or risk and the level of differential privacy guaranteed by mechanisms for statistical query processing. In this paper we study this trade-off in private Support Vector Machine (SVM) learning. We present two efficient mechanisms, one for the case of finite-dimensional feature mappings and one for potentially infinite-dimensional feature mappings with translation-invariant kernels. For the case of translation-invariant kernels, the proposed mechanism minimizes regularized empirical risk in a random Reproducing Kernel Hilbert Space whose kernel uniformly approximates the desired kernel with high probability. This technique, borrowed from large-scale learning, allows the mechanism to respond with a finite encoding of the classifier, even when the function class is of infinite VC dimension. Differential privacy is established using a proof technique from algorithmic stability. Utility–the mechanism’s response function is pointwise epsilon-close to non-private SVM with probability 1-delta–is proven by appealing to the smoothness of regularized empirical risk minimization with respect to small perturbations to the feature mapping. We conclude with a lower bound on the optimal differential privacy of the SVM. This negative result states that for any delta, no mechanism can be simultaneously (epsilon,delta)-useful and beta-differentially private for small epsilon and small beta.

💡 Research Summary

This paper investigates the fundamental trade‑off between utility and privacy in the context of Support Vector Machine (SVM) learning under differential privacy. While prior work has largely focused on private statistical query answering, the authors address the more complex problem of privately training an SVM, which involves solving a regularized empirical risk minimization (RERM) problem in a potentially high‑ or infinite‑dimensional feature space. The contribution consists of two efficient mechanisms, each tailored to a different class of kernels, together with rigorous privacy and utility analyses and a lower‑bound result that delineates the limits of what can be achieved.

1. Problem Setting and Notation
The authors consider a standard binary classification setting with training data D = {(x_i, y_i)}_{i=1}^n, where x_i ∈ X and y_i ∈ {−1, +1}. An SVM learns a weight vector w that minimizes the regularized hinge loss

  R_D(w) = (1/n) Σ_i ℓ(y_i·⟨w, φ(x_i)⟩) + (λ/2)‖w‖²,

where φ: X → H is a feature map into a reproducing kernel Hilbert space (RKHS) associated with kernel k, λ > 0 is the regularization parameter, and ℓ(z) = max(0, 1−z) is the hinge loss. The goal is to release a classifier ŷ(x) = sign(⟨ŵ, φ(x)⟩) that is (β‑differentially) private while remaining close to the non‑private solution w*.

2. Finite‑Dimensional Feature Maps
When φ maps into a finite‑dimensional Euclidean space ℝ^d, the authors propose a mechanism based on algorithmic stability rather than classical global sensitivity. They first show that the RERM objective is λ‑strongly convex, which implies that the solution mapping D ↦ w_D is β‑stable: changing a single training example changes w_D by at most (2L)/(λ n) in ℓ₂ norm, where L bounds the loss gradient. By adding Gaussian noise with variance proportional to this stability bound, they achieve (β, δ)‑differential privacy. The key insight is that stability yields a data‑dependent sensitivity that can be much smaller than the worst‑case bound, allowing the noise magnitude to shrink as n grows.

Utility is analyzed by exploiting the smoothness of the regularized risk: the loss function is Lipschitz in its argument, and the regularizer is smooth. Consequently, the perturbed solution Ŵ satisfies

  |R_D(Ŵ) – R_D(w*)| ≤ O( (L²)/(λ n) + σ·√(d) ),

where σ is the standard deviation of the added Gaussian noise. By choosing σ = Θ(β/(λ n)), the excess risk scales as O(1/(λ n) + β/(λ n)·√d). This yields a pointwise guarantee: with probability at least 1−δ, for every x the private classifier’s decision value differs from the non‑private one by at most ε.

3. Translation‑Invariant Kernels and Random Feature Approximation
The second mechanism tackles kernels of the form k(x, x′) = ψ(x−x′), which induce infinite‑dimensional RKHSs (e.g., Gaussian RBF). Directly applying the finite‑dimensional approach is impossible because the feature map is not explicit. The authors adopt Random Fourier Features (RFF), a technique introduced by Rahimi and Recht, to construct an explicit low‑dimensional approximation Φ_D: X → ℝ^D. By sampling D frequencies ω₁,…,ω_D from the spectral distribution p(ω) (the Fourier transform of ψ) and random phases b_j ∼ Uniform