A Certified Unlearning Approach without Access to Source Data

With the growing adoption of data privacy regulations, the ability to erase private or copyrighted information from trained models has become a crucial requirement. Traditional unlearning methods often assume access to the complete training dataset, which is unrealistic in scenarios where the source data is no longer available. To address this challenge, we propose a certified unlearning framework that enables effective data removal \final{without access to the original training data samples}. Our approach utilizes a surrogate dataset that approximates the statistical properties of the source data, allowing for controlled noise scaling based on the statistical distance between the two. \updated{While our theoretical guarantees assume knowledge of the exact statistical distance, practical implementations typically approximate this distance, resulting in potentially weaker but still meaningful privacy guarantees.} This ensures strong guarantees on the model’s behavior post-unlearning while maintaining its overall utility. We establish theoretical bounds, introduce practical noise calibration techniques, and validate our method through extensive experiments on both synthetic and real-world datasets. The results demonstrate the effectiveness and reliability of our approach in privacy-sensitive settings.

💡 Research Summary

The paper tackles a pressing problem in machine learning privacy: how to perform certified unlearning when the original training data are no longer available. Certified unlearning traditionally guarantees that a model after deletion of a subset of data behaves indistinguishably from a model retrained from scratch on the remaining data, usually by bounding the statistical distance between the two models and adding calibrated Gaussian noise. Existing methods, however, assume full access to the original dataset to compute gradients, Hessians, or other statistics required for the update.

To overcome this limitation, the authors introduce a surrogate‑data framework. A surrogate dataset (D_s) is a collection of samples that share the same feature‑label support as the original data but are drawn from a possibly different distribution (\nu). The key insight is that the statistical distance between the true data distribution (\rho) and the surrogate distribution (\nu) can be used to control the amount of noise needed for certification. Specifically, they focus on the total variation distance (TV(\rho|\nu)) as a measure of discrepancy.

The methodology consists of four steps:

1. Hessian Estimation – Since the Hessian of the retained data (H_{D_r}) is unavailable, the algorithm computes the Hessian on the surrogate data (H_{D_s}) and adjusts it using the known size of the forget set (m) and the total dataset size (n). The adjustment follows a linear correction: (\hat{H}{D_r}= \frac{n}{n-m} H{D_s} - \frac{m}{n-m} H_{D_u}), where (H_{D_u}) is the Hessian of the forget set (estimated if not present in the surrogate).

2. Model Update – With the estimated Hessian, a single‑step Newton update is performed on the original model parameters (w^\ast): (\tilde{w}= w^\ast - \hat{H}_{D_r}^{-1}\nabla L(D_r,w^\ast)). The gradient (\nabla L(D_r,w^\ast)) is similarly approximated using surrogate statistics.

3. Noise Calibration – The authors prove (Theorem 4.1) that the Euclidean distance between the truly retrained model (w^\ast_r) and the approximated model (\tilde{w}) is bounded by a function (\Delta) that scales linearly with (TV(\rho|\nu)) and depends on the loss’s Lipschitz, strong‑convexity, smoothness, and Hessian‑Lipschitz constants.

4. Gaussian Mechanism – A Gaussian noise vector (n\sim\mathcal{N}(0,\sigma^2 I)) is added to (\tilde{w}) to obtain the final unlearned model (\hat{w}= \tilde{w}+n). The variance (\sigma^2) is set according to the standard differential‑privacy conversion: (\sigma^2 = \frac{2\ln(1.25/\delta)}{\varepsilon^2}\Delta^2). This guarantees ((\varepsilon,\delta))-certified unlearning as defined in prior work (Sekhari et al., 2021).

The theoretical contribution is twofold: (i) establishing a rigorous bound that directly ties the certification parameters to the statistical distance between source and surrogate distributions, and (ii) showing that even when the exact distance is unknown, a model‑based estimator (e.g., using MMD or KL approximations) can provide a usable upper bound, albeit with slightly weaker guarantees.

Empirical validation is performed on synthetic Gaussian mixtures and real‑world image benchmarks (MNIST, CIFAR‑10). Surrogate datasets are constructed from different sources (e.g., Fashion‑MNIST for MNIST, STL‑10 for CIFAR‑10) while preserving class proportions. The experiments demonstrate:

• When (TV(\rho|\nu)) is small (≈0.1–0.2), the calibrated noise is modest, and the utility loss (accuracy drop) is comparable to methods that have full data access (typically <1%).
• As the surrogate becomes less representative (higher TV), the required (\varepsilon) grows, but the method still satisfies the certified guarantee, confirming the robustness of the framework.
• The proposed distance‑estimation heuristic, which relies only on the current model and surrogate data, yields ε values within 20 % of the oracle case where the true TV is known.

Key strengths of the work include:

• Practical relevance – Many production systems delete raw data for compliance or storage reasons; this paper provides a concrete, provably secure pathway to still honor delete requests.
• Modular design – The approach plugs into any existing single‑step Newton‑based unlearning pipeline, requiring only a surrogate dataset and a distance estimator.
• Clear theoretical‑experimental bridge – The authors derive explicit formulas linking statistical distance to noise scale, and then validate them empirically.

Limitations and open questions are also acknowledged:

• Computing or approximating the Hessian for large deep networks remains costly; future work could explore low‑rank or Kronecker‑factored approximations.
• The total variation distance may be difficult to estimate accurately in high dimensions; alternative divergences (Wasserstein, Rényi) could provide tighter or more tractable bounds.
• The current experiments focus on moderate‑size vision datasets; scaling to transformer‑scale language models is an important next step.

In summary, the paper presents the first certified unlearning framework that operates without any access to the original training data, leveraging a surrogate dataset and a statistically grounded noise calibration. By tying the certification parameters to a measurable distance between source and surrogate distributions, it offers both strong privacy guarantees and practical utility, opening a viable path for privacy‑compliant model maintenance in real‑world settings.