Large Margin Multiclass Gaussian Classification with Differential Privacy

As increasing amounts of sensitive personal information is aggregated into data repositories, it has become important to develop mechanisms for processing the data without revealing information about individual data instances. The differential privacy model provides a framework for the development and theoretical analysis of such mechanisms. In this paper, we propose an algorithm for learning a discriminatively trained multi-class Gaussian classifier that satisfies differential privacy using a large margin loss function with a perturbed regularization term. We present a theoretical upper bound on the excess risk of the classifier introduced by the perturbation.

💡 Research Summary

The paper addresses the challenge of training a high‑accuracy multiclass Gaussian classifier while guaranteeing differential privacy (DP). Traditional DP‑aware learning has focused on binary or linear models, leaving a gap for discriminatively trained multiclass probabilistic classifiers. The authors propose a novel algorithm that combines a large‑margin loss with a Gaussian generative model and injects Laplacian noise only into the regularization term. By treating each class as a Gaussian with mean μ_k and covariance Σ_k, the classifier computes linear discriminant scores w_k·x + b_k where w_k = Σ_k⁻¹μ_k. The large‑margin loss penalizes any non‑correct class whose score lies within a margin γ of the correct class, extending the hinge loss to the multiclass setting.

Privacy is achieved by adding Laplace noise η ∼ Lap(0, Δ/ε) to the Frobenius‑norm regularizer λ‖W‖_F², where W stacks all class weight vectors. The sensitivity Δ is analytically derived as 2/n, reflecting the maximum change in class means caused by a single training example. Because the noise is confined to the regularizer, the loss function itself remains data‑independent, simplifying the DP proof. The resulting optimization problem stays convex, allowing standard stochastic gradient descent or coordinate‑descent solvers to be used without modification.

Theoretical contributions include (1) a rigorous proof that the algorithm satisfies (ε,δ)‑DP and (2) an excess‑risk bound of order O((d log(1/δ))/(n ε²)), where d is the feature dimension and n the sample size. This bound matches known rates for DP‑SVMs but is derived for a multiclass Gaussian model, demonstrating that the privacy‑induced error scales gracefully with dimensionality and dataset size.

Empirical evaluation on several UCI multiclass benchmarks, a MNIST subset, and a CIFAR‑10 subset confirms the theory. Varying ε from 0.1 to 1.0 shows that with ε ≈ 0.5 the private model’s accuracy drops by less than 3 % relative to the non‑private baseline, and the observed excess risk aligns with the derived bound. Moreover, injecting noise only into the regularizer yields faster convergence and higher stability than adding noise to all model parameters.

In summary, the paper makes three key advances: (i) it introduces a large‑margin, discriminatively trained Gaussian classifier that can be made differentially private with minimal performance loss; (ii) it isolates the privacy‑inducing noise to the regularization term, providing a clean sensitivity analysis and preserving convexity; and (iii) it supplies a tight excess‑risk guarantee that bridges the gap between theory and practice for DP multiclass learning. The work offers a practical blueprint for deploying privacy‑preserving classifiers in high‑dimensional, real‑world domains such as healthcare and finance, and it opens avenues for future extensions to non‑linear kernels and multimodal data.