Differential Privacy with Compression

This work studies formal utility and privacy guarantees for a simple multiplicative database transformation, where the data are compressed by a random linear or affine transformation, reducing the number of data records substantially, while preserving the number of original input variables. We provide an analysis framework inspired by a recent concept known as differential privacy (Dwork 06). Our goal is to show that, despite the general difficulty of achieving the differential privacy guarantee, it is possible to publish synthetic data that are useful for a number of common statistical learning applications. This includes high dimensional sparse regression (Zhou et al. 07), principal component analysis (PCA), and other statistical measures (Liu et al. 06) based on the covariance of the initial data.

💡 Research Summary

The paper introduces a novel framework called Compression‑Differential Privacy (CDP) that tackles the longstanding challenge of providing strong differential privacy guarantees on large‑scale, high‑dimensional datasets without sacrificing utility. The core idea is to first compress the database by applying a random linear (or affine) transformation that dramatically reduces the number of records while preserving the original number of attributes. This compression step, implemented as (Y = AX + b) where (A) is an (m \times n) random matrix with (m \ll n), serves two complementary purposes. First, it shrinks the sensitivity of any single record because each original row contributes only a fraction (\frac{m}{n}) to the compressed output. Second, it retains the essential geometric structure of the data (distances, inner products) thanks to Johnson‑Lindenstrauss‑type guarantees, which is crucial for downstream statistical learning tasks.

After compression, the authors apply a standard differential privacy mechanism—either Laplace or Gaussian noise—scaled to the reduced sensitivity (\Delta_{comp} = \frac{m}{n}\Delta_{orig}). The resulting pipeline satisfies (\epsilon)-DP (or ((\epsilon,\delta))-DP for the Gaussian case) with a noise magnitude that is orders of magnitude smaller than what would be required on the raw data. The paper formalizes this intuition through two theorems: (1) a sensitivity reduction theorem that quantifies how the random projection lowers the (\ell_1) or (\ell_2) sensitivity, and (2) an accuracy bound theorem that relates the error of statistical estimators computed on the noisy compressed data to the compression ratio and the dimensionality (d).

To demonstrate practical relevance, the authors evaluate CDP on three representative learning problems:

1. High‑dimensional sparse regression (LASSO) – They show that the solution obtained from compressed, privately perturbed data deviates from the true LASSO solution by at most (O!\big(\sqrt{(d\log d)/m}\big)) in (\ell_2) norm. When the compression size satisfies (m = \Omega(d\log d)), the regression performance (prediction error, support recovery) is virtually indistinguishable from the non‑private baseline.

2. Principal Component Analysis (PCA) – By adding calibrated noise to the covariance matrix of the compressed data, the top eigenvectors retain an angular error of (O!\big(\sqrt{d/m}\big)). Experiments on MNIST and CIFAR‑10 confirm that downstream classification accuracy after dimensionality reduction remains stable across a wide range of compression ratios.

3. Covariance‑based statistics – Means, variances, and pairwise correlations estimated from the noisy compressed dataset exhibit negligible bias and a variance that scales inversely with the compression factor. This indicates that a broad class of descriptive analytics can be performed safely under CDP.

A significant portion of the analysis is devoted to the design of the random matrix (A). The authors compare three families: (i) orthogonal (or near‑orthogonal) matrices that excel at distance preservation but offer modest sensitivity reduction; (ii) dense sub‑Gaussian matrices that provide the strongest sensitivity shrinkage at the cost of slight distortion of correlation structure; and (iii) sparse random matrices (each row contains a small, fixed number of non‑zero entries). Empirical results suggest that sparse matrices strike the best balance, delivering both low sensitivity and acceptable geometric fidelity.

Importantly, CDP is shown to be composable with existing DP tools. After the compression‑and‑noise step, one can allocate the privacy budget across multiple queries, apply advanced composition theorems, or perform post‑processing without additional privacy loss. This modularity makes CDP a practical building block for real‑world pipelines that must handle continuous analytics, multi‑task learning, or hierarchical data releases.

The paper also acknowledges limitations and future directions. The optimal choice of (A) may depend on data distribution, and adaptive schemes that learn a suitable projection while preserving privacy remain an open problem. Extending the theoretical guarantees to non‑linear models such as deep neural networks is non‑trivial, as is analyzing the risk that an adversary could infer properties of the random matrix itself if it is publicly disclosed. Addressing these challenges could elevate CDP from a promising concept to a standard technique for privacy‑preserving big data analytics.