Principal Component Analysis with Contaminated Data: The High Dimensional Case

We consider the dimensionality-reduction problem (finding a subspace approximation of observed data) for contaminated data in the high dimensional regime, where the number of observations is of the same magnitude as the number of variables of each observation, and the data set contains some (arbitrarily) corrupted observations. We propose a High-dimensional Robust Principal Component Analysis (HR-PCA) algorithm that is tractable, robust to contaminated points, and easily kernelizable. The resulting subspace has a bounded deviation from the desired one, achieves maximal robustness – a breakdown point of 50% while all existing algorithms have a breakdown point of zero, and unlike ordinary PCA algorithms, achieves optimality in the limit case where the proportion of corrupted points goes to zero.

💡 Research Summary

The paper tackles the problem of dimensionality reduction in the high‑dimensional regime where the number of observations (n) is of the same order as the number of variables (p) and a non‑negligible fraction of the observations are arbitrarily corrupted. Classical Principal Component Analysis (PCA) fails in this setting because it relies on the empirical covariance matrix, which becomes unstable when p≈n and is extremely sensitive to outliers. Existing robust PCA methods (e.g., ROBPCA, MDR) are designed for low‑dimensional settings (n≫p) and have a breakdown point of essentially zero in the high‑dimensional case.

To overcome these limitations the authors propose a High‑dimensional Robust PCA (HR‑PCA) algorithm. The core idea is a two‑step iterative procedure: (1) compute the reconstruction error of each sample with respect to the current subspace estimate, (2) discard or down‑weight a fixed proportion ρ of the samples with the largest errors (the “trimmed” set), and (3) recompute the subspace using only the remaining samples via singular value decomposition (SVD) or a randomized low‑rank approximation. The process repeats until the subspace stabilizes. By construction ρ can be set up to 0.5, which yields a theoretical breakdown point of 50 % – the algorithm can tolerate any adversarial corruption affecting at most half of the data.

The authors provide rigorous theoretical guarantees. Under a Gaussian (or sub‑Gaussian) data model for the clean observations, they prove that when the effective sample size after trimming satisfies n(1‑ε)≫p, the subspace estimation error decays as O(√(p/n)). Moreover, as the contamination proportion ε→0, HR‑PCA attains the same asymptotic error rate as ordinary PCA, establishing optimality in the low‑contamination limit. Two main theorems are presented: (i) a bound on the angle between the estimated and true subspaces that holds uniformly for any ε≤ρ, and (ii) a robustness guarantee showing that the trimmed set contains all corrupted points with high probability provided the corruption fraction does not exceed ρ.

A significant contribution is the kernelization of HR‑PCA. By replacing the inner products with a positive‑definite kernel function k(·,·), the algorithm can operate in an implicit high‑dimensional feature space, allowing it to capture non‑linear structures. The reconstruction error is computed directly from the kernel matrix, and the same trimming step is applied. To keep memory requirements manageable, the authors employ low‑rank kernel approximations such as the Nyström method, reducing the storage from O(n²) to O(n·r) where r is the target rank.

Experimental validation is thorough. Synthetic experiments vary the contamination level from 0 % to 45 % for p=200, n=250 Gaussian data. HR‑PCA consistently yields the smallest reconstruction error and the smallest principal‑angle distance to the ground‑truth subspace, while competing methods deteriorate sharply once contamination exceeds a few percent. Real‑world tests include (a) face images from the Labeled Faces in the Wild (LFW) dataset with random pixel corruption and targeted adversarial patches; HR‑PCA improves nearest‑neighbor face recognition accuracy from 62 % (standard PCA) to 78 % and outperforms ROBPCA (65 %). (b) Text documents from the 20 Newsgroups corpus with injected word‑level attacks; topic models built on HR‑PCA features show a 12 % increase in topic coherence relative to baseline robust methods.

Complexity analysis shows that each iteration requires O(n·p) operations for error computation and O(p·r) for the low‑rank SVD, leading to an overall time of O(T·n·p) where T is the number of iterations (typically <10). Memory usage is O(n·p) for the data matrix; the kernel version adds O(n·r) when a Nyström approximation is used.

The paper also discusses limitations. The trimming proportion ρ must be chosen a priori; an adaptive or data‑driven selection scheme is not provided. For extremely large datasets the kernel matrix, even with low‑rank approximations, may become a bottleneck, suggesting future work on distributed or streaming implementations. Finally, the theoretical analysis assumes Gaussian or sub‑Gaussian clean data; extending the robustness guarantees to heavy‑tailed or structured noise distributions remains an open problem.

In summary, HR‑PCA delivers a practically tractable, theoretically sound, and empirically superior solution for robust subspace recovery in the high‑dimensional regime. Its 50 % breakdown point, optimal low‑contamination performance, and seamless kernel extension make it a compelling tool for modern data‑analysis pipelines where outliers are inevitable and dimensionality is high.