Less Noise, Same Certificate: Retain Sensitivity for Unlearning

Certified machine unlearning aims to provably remove the influence of a deletion set $U$ from a model trained on a dataset $S$, by producing an unlearned output that is statistically indistinguishable from retraining on the retain set $R:=S\setminus U$. Many existing certified unlearning methods adapt techniques from Differential Privacy (DP) and add noise calibrated to global sensitivity, i.e., the worst-case output change over all adjacent datasets. We show that this DP-style calibration is often overly conservative for unlearning, based on a key observation: certified unlearning, by definition, does not require protecting the privacy of the retained data $R$. Motivated by this distinction, we define retain sensitivity as the worst-case output change over deletions $U$ while keeping $R$ fixed. While insufficient for DP, retain sensitivity is exactly sufficient for unlearning, allowing for the same certificates with less noise. We validate these reductions in noise theoretically and empirically across several problems, including the weight of minimum spanning trees, PCA, and ERM. Finally, we refine the analysis of two widely used certified unlearning algorithms through the lens of retain sensitivity, leveraging the regularity induced by $R$ to further reduce noise and improve utility.

💡 Research Summary

The paper addresses a fundamental inefficiency in certified machine unlearning: the reliance on differential‑privacy (DP) techniques that calibrate noise to the global sensitivity (GS) of a learning algorithm. GS measures the worst‑case change in the algorithm’s output over all possible neighboring datasets, and consequently forces the addition of a large amount of noise to guarantee that the unlearned model is indistinguishable from a model retrained on the retained set R = S \ U. However, certified unlearning does not need to protect the privacy of the retained data R; it only needs to hide the influence of the deletion set U. This observation motivates the introduction of a new data‑dependent quantity called retain sensitivity (RS).

Definition of RS. For a fixed retained dataset R, RS is the maximum ℓ₂‑distance between the algorithm’s output on R and on R ∪ {z}, where z is any single additional record (the potential deletion). Formally, RS_f(R) = max_{z} ‖f(R ∪ {z}) – f(R)‖. By construction RS ≤ LS ≤ GS, where LS is the usual local sensitivity. Unlike LS, RS depends only on R and not on the unknown deletion set U, making it sufficient for unlearning while being unnecessary for DP.

Core theoretical result. The authors prove (Theorem 2.11) that for any learning–unlearning pair (A, Ā) that adds isotropic Gaussian noise ν ∼ N(0,σ²I), setting σ = RS(A, Ā)(R)·ε⁻¹·√(2 log(1.25/δ)) yields an (ε, δ)‑unlearning guarantee. The proof hinges on the fact that both the “delete U then unlearn” and the “retrain on R” executions share the same retained core R, so the same noise distribution can be used on both sides of the indistinguishability test. In contrast, DP must compare arbitrary neighboring datasets that may share no common core, which forces the use of the larger GS.

Applications to canonical problems. The paper derives RS bounds for several standard tasks, showing that RS can be orders of magnitude smaller than GS:

1. Median. GS = B/2 for data bounded in