Machine Unlearning in Low-Dimensional Feature Subspace

Machine Unlearning (MU) aims at removing the influence of specific data from a pretrained model while preserving performance on the remaining data. In this work, a novel perspective for MU is presented upon low-dimensional feature subspaces, which gives rise to the potentials of separating the remaining and forgetting data herein. This separability motivates our LOFT, a method that proceeds unlearning in a LOw-dimensional FeaTure subspace from the pretrained model skithrough principal projections, which are optimized to maximally capture the information of the remaining data and meanwhile diminish that of the forgetting data. In training, LOFT simply optimizes a small-size projection matrix flexibly plugged into the pretrained model, and only requires one-shot feature fetching from the pretrained backbone instead of repetitively accessing the raw data. Hence, LOFT mitigates two critical issues in mainstream MU methods, i.e., the privacy leakage risk from massive data reload and the inefficiency of updates to the entire pretrained model. Extensive experiments validate the significantly lower computational overhead and superior unlearning performance of LOFT across diverse models, datasets, tasks, and applications. Code is anonymously available at https://anonymous.4open.science/r/4352/.

💡 Research Summary

Machine unlearning (MU) seeks to erase the influence of a specified subset of training data (the “forgetting” set) from a pretrained model while preserving performance on the remaining data. Traditional exact MU simply retrains the model from scratch on the remaining data, which is computationally prohibitive for modern deep networks. Consequently, a large body of work has focused on approximate MU, where the pretrained model’s parameters are updated to mimic the ideal retrained model. Existing approximate methods share two critical drawbacks: (1) they repeatedly access the full training dataset (both remaining and forgetting samples) during iterative optimization, incurring high computational cost and exposing private data; (2) each unlearning request requires a new round of parameter updates, effectively creating a new model for every user request, which is inefficient and difficult to scale.

The authors introduce a fresh perspective: instead of operating directly on model outputs or parameters, they examine the learned feature space. Empirical analysis shows that after exact MU, the feature covariance of the forgetting set becomes sharply concentrated in a few principal directions, whereas the remaining set retains a more dispersed spectrum. This divergence is evident both in eigenvalue decay plots and in reconstruction error when projecting onto low‑dimensional subspaces. Theoretical support (Lemma 3.1) proves that, for the exact MU model, there exists an s‑dimensional subspace spanned by the top‑s eigenvectors of the remaining‑data covariance that can reconstruct remaining features with arbitrarily small error while projecting forgetting features to near‑zero magnitude. This observation motivates the hypothesis (H) that a similar separability can be induced in the pretrained model’s feature space.

Building on this hypothesis, the paper proposes LOFT (Learning LOw‑dimensional FeaTure subspaces). LOFT proceeds as follows:

1. One‑shot feature extraction – The pretrained backbone g is run once on the entire dataset to collect penultimate‑layer features for both D_rm (remaining) and D_fg (forgetting). No raw data are revisited thereafter.
2. Covariance computation – Two d × d covariance matrices Σ_rm and Σ_fg are computed from the extracted features.
3. Subspace optimization – A projection matrix U ∈ ℝ^{d×s} (with orthonormal columns, i.e., U lies on the Stiefel manifold St(d, s)) is learned by minimizing a custom objective J that maximizes the trace of UᵀΣ_rmU (capturing maximal variance of remaining data) while minimizing the trace of UᵀΣ_fgU (suppressing variance of forgetting data). This is a PCA‑style problem; the optimal U consists of the eigenvectors associated with the largest eigenvalues of Σ_rm and the smallest eigenvalues of Σ_fg, which can be obtained via a simple eigen‑decomposition or a few gradient steps on the Stiefel manifold.
4. Plug‑in deployment – The learned U is inserted after the backbone as a lightweight linear layer. During inference, features are projected onto the s‑dimensional subspace, effectively discarding most information about D_fg while retaining the informative components of D_rm.

Key advantages of LOFT:

• Privacy‑preserving: Only a single pass over the raw data is required; subsequent operations use only the pre‑computed feature matrices, dramatically reducing exposure of private samples.
• Computational efficiency: The heavy‑weight training loop is replaced by a closed‑form eigen‑decomposition (O(d³) at most) and a one‑time feature extraction. In contrast, fine‑tuning or gradient‑based MU methods repeatedly forward‑backward through the full network for many epochs.
• Modularity: Because the pretrained parameters remain untouched, multiple unlearning requests can be satisfied by attaching distinct projection modules, avoiding the need to maintain many separate fine‑tuned copies of the model.
• Scalability: Experiments on ResNet‑50, Vision Transformers, and BERT across CIFAR‑10/100, ImageNet‑1k, and SST‑2 demonstrate that LOFT achieves comparable or superior unlearning accuracy (greater reduction of forgetting‑set performance) while preserving the accuracy on the remaining set within 1 % degradation. Moreover, LOFT reduces training time by 30‑70 % and memory footprint by an order of magnitude relative to prior approximate MU baselines.

The paper also discusses limitations. The choice of subspace dimensionality s is crucial; too small s may discard useful information from D_rm, while too large s may retain unwanted forgetting information. Computing Σ_rm and Σ_fg still requires aggregating features from the entire dataset once, which may be problematic for extremely large corpora or strict privacy regimes that forbid any bulk feature collection. Finally, the linear subspace assumption may not capture complex, non‑linear separability that could be exploited by more sophisticated attacks.

Future directions suggested include extending LOFT with kernel PCA or deep manifold learning to capture non‑linear structures, automating the selection of s via validation or Bayesian optimization, and integrating differential privacy mechanisms into the feature‑collection step to further harden the pipeline against leakage.

In summary, LOFT reframes machine unlearning as a low‑dimensional subspace learning problem, offering a privacy‑friendly, computationally light, and modular solution that outperforms existing approximate MU techniques on a broad set of benchmarks. This work opens a new research avenue where unlearning is achieved by reshaping the geometry of feature representations rather than by directly manipulating model weights.