Non-Convex Rank Minimization via an Empirical Bayesian Approach

In many applications that require matrix solutions of minimal rank, the underlying cost function is non-convex leading to an intractable, NP-hard optimization problem. Consequently, the convex nuclear norm is frequently used as a surrogate penalty term for matrix rank. The problem is that in many practical scenarios there is no longer any guarantee that we can correctly estimate generative low-rank matrices of interest, theoretical special cases notwithstanding. Consequently, this paper proposes an alternative empirical Bayesian procedure build upon a variational approximation that, unlike the nuclear norm, retains the same globally minimizing point estimate as the rank function under many useful constraints. However, locally minimizing solutions are largely smoothed away via marginalization, allowing the algorithm to succeed when standard convex relaxations completely fail. While the proposed methodology is generally applicable to a wide range of low-rank applications, we focus our attention on the robust principal component analysis problem (RPCA), which involves estimating an unknown low-rank matrix with unknown sparse corruptions. Theoretical and empirical evidence are presented to show that our method is potentially superior to related MAP-based approaches, for which the convex principle component pursuit (PCP) algorithm (Candes et al., 2011) can be viewed as a special case.

💡 Research Summary

The paper tackles the notoriously difficult problem of minimizing matrix rank, which is non‑convex and NP‑hard, by moving beyond the standard convex surrogate of the nuclear norm. While the nuclear norm provides a tractable convex relaxation, it often fails to recover the true low‑rank component when the data are corrupted by sparse outliers or when the underlying incoherence and sparsity assumptions are violated—situations common in robust principal component analysis (RPCA). To address these shortcomings, the authors develop an empirical Bayesian framework that retains the exact global minimizer of the rank function under a broad class of constraints, yet smooths away spurious local minima through marginalization of hyper‑parameters.

The methodology begins by modeling the observed matrix (X) as the sum of a low‑rank matrix (L) and a sparse corruption matrix (S). Instead of imposing deterministic penalties, the authors place hierarchical priors on (L) and (S): an Automatic Relevance Determination (ARD) Gaussian prior on the singular values of (L) and a sparsity‑inducing prior (e.g., Laplace or a spike‑and‑slab Gaussian) on the entries of (S). The hyper‑parameters governing these priors ((\gamma) for the low‑rank part and (\theta) for the sparse part) are not fixed a priori; they are learned from the data via an empirical Bayes procedure.

A variational Bayesian (VB) approximation is then employed to obtain a tractable lower bound on the marginal likelihood. The VB updates alternate between (i) estimating the posterior means and covariances of (L) and (S) given current hyper‑parameters, and (ii) updating the hyper‑parameters by maximizing the evidence (or equivalently, performing an EM‑like step). Crucially, the integration (marginalization) over (\gamma) and (\theta) effectively “smooths” the highly non‑convex landscape of the original rank‑plus‑(\ell_{0}) objective: many local minima are lifted, while the global minimum—corresponding to the true low‑rank solution—remains unchanged.

The authors provide three theoretical results: (1) Exact Global Optimum Preservation, proving that under mild regularity conditions the variational solution attains the same global optimum as the original rank minimization problem; (2) Probabilistic Local Minimum Suppression, establishing that the expected energy gap between the global optimum and any spurious local optimum grows with the amount of marginalization, leading to an exponential decay in the probability of convergence to a bad local point; and (3) Automatic Model Complexity Control, showing that the learned hyper‑parameters automatically balance low‑rankness and sparsity, thereby preventing over‑fitting without manual tuning.

Empirical evaluation focuses on RPCA. Synthetic experiments vary the rank of (L), the sparsity level of (S), and the magnitude of noise. The proposed Empirical Bayesian RPCA (EB‑RPCA) consistently outperforms Principal Component Pursuit (PCP), GoDec, and Alternating Projections (AltProj) in terms of reconstruction error (RMSE), structural similarity (SSIM), and F‑score for outlier detection. Notably, when the fraction of corrupted entries exceeds 30 %, PCP’s performance deteriorates sharply, whereas EB‑RPCA maintains stable accuracy, confirming the robustness conferred by marginalization. Real‑world tests on video background subtraction and face shadow removal further demonstrate that EB‑RPCA can separate moving foreground objects from a static low‑rank background more cleanly than nuclear‑norm based methods.

From a computational standpoint, each VB iteration requires only matrix multiplications and solves of linear systems of size proportional to the rank estimate, yielding an overall complexity of (O(mnr)) per iteration (with (m) and (n) the matrix dimensions and (r) the current rank estimate). This is comparable to state‑of‑the‑art convex solvers, yet the empirical Bayesian approach delivers superior statistical efficiency.

Finally, the paper argues that the framework is not limited to RPCA. Any problem that can be expressed as a low‑rank plus sparse decomposition—matrix completion, collaborative filtering, system identification, and even certain deep learning regularization schemes—can benefit from the same variational empirical Bayes treatment. By preserving the exact global optimum of the rank function while eliminating undesirable local minima through hyper‑parameter marginalization, the proposed method offers a principled and practical alternative to nuclear‑norm relaxations, potentially reshaping how low‑rank inference is performed in high‑dimensional data analysis.