Guaranteed Rank Minimization via Singular Value Projection

Minimizing the rank of a matrix subject to affine constraints is a fundamental problem with many important applications in machine learning and statistics. In this paper we propose a simple and fast algorithm SVP (Singular Value Projection) for rank minimization with affine constraints (ARMP) and show that SVP recovers the minimum rank solution for affine constraints that satisfy the “restricted isometry property” and show robustness of our method to noise. Our results improve upon a recent breakthrough by Recht, Fazel and Parillo (RFP07) and Lee and Bresler (LB09) in three significant ways: 1) our method (SVP) is significantly simpler to analyze and easier to implement, 2) we give recovery guarantees under strictly weaker isometry assumptions 3) we give geometric convergence guarantees for SVP even in presense of noise and, as demonstrated empirically, SVP is significantly faster on real-world and synthetic problems. In addition, we address the practically important problem of low-rank matrix completion (MCP), which can be seen as a special case of ARMP. We empirically demonstrate that our algorithm recovers low-rank incoherent matrices from an almost optimal number of uniformly sampled entries. We make partial progress towards proving exact recovery and provide some intuition for the strong performance of SVP applied to matrix completion by showing a more restricted isometry property. Our algorithm outperforms existing methods, such as those of \cite{RFP07,CR08,CT09,CCS08,KOM09,LB09}, for ARMP and the matrix-completion problem by an order of magnitude and is also significantly more robust to noise.

💡 Research Summary

The paper tackles the Affine Rank Minimization Problem (ARMP), which seeks the lowest‑rank matrix X satisfying a set of linear (affine) constraints 𝒜(X)=b. This problem underlies many tasks such as system identification, collaborative filtering, and compressed sensing for matrices. Traditional approaches replace the non‑convex rank objective with the convex nuclear‑norm surrogate, but these methods are computationally heavy and require very strong restricted isometry property (RIP) conditions for exact recovery.

The authors introduce Singular Value Projection (SVP), an algorithm that alternates between a gradient‑type update and a hard rank‑r projection. At iteration k, the residual r_k = b – 𝒜(X_k) is computed, and a step X_k′ = X_k + μ·𝒜* (r_k) is taken, where 𝒜* is the adjoint of 𝒜 and μ is a step‑size. Then X_{k+1} = P_r (X_k′) is obtained by performing a singular value decomposition of X_k′ and keeping only the top r singular values, zero‑ing out the rest. This projection enforces the rank constraint directly, rather than encouraging low rank indirectly via a regularizer.

The theoretical contribution is threefold. First, the authors prove that SVP exactly recovers the minimum‑rank solution whenever the measurement operator 𝒜 satisfies a rank‑2r restricted isometry constant δ_{2r} < 1/3. This is a substantially weaker requirement than the δ_{2r} < 0.1 bound needed in earlier works (e.g., Recht‑Fazel‑Parrilo 2007, Lee‑Bresler 2009). The proof hinges on showing that each SVP iteration reduces the distance to the true solution by a factor ρ < 1, provided the RIP holds.

Second, the analysis extends to noisy observations b = 𝒜(X*) + e. In this setting the authors establish a linear error bound: ‖X_k – X*‖_F ≤ ρ^k‖X_0 – X*‖_F + C·‖e‖_2, where C depends only on the RIP constants. Consequently, the algorithm converges geometrically to a neighborhood whose radius is proportional to the noise level, demonstrating robustness.

Third, the paper applies SVP to the matrix completion problem, a special case of ARMP where 𝒜 samples individual entries. Under the standard incoherence and uniform sampling assumptions, the authors show empirically that SVP recovers low‑rank matrices from as few as O(n r log n) observed entries—essentially the information‑theoretic optimum. They also discuss a more restrictive “matrix RIP” that holds for random sampling and underpins the observed performance.

From a computational standpoint, each SVP iteration requires only the top‑r singular vectors of an m×n matrix. Using Lanczos or power‑method approximations, the per‑iteration cost is O(r·m·n), and the total cost to achieve ε‑accuracy is O(r·m·n·log(1/ε)). This is dramatically lower than semidefinite programming or augmented‑Lagrangian schemes, which scale at least quadratically in the matrix dimensions.

The experimental section validates the theory on both synthetic and real data. Synthetic tests vary the RIP constant, rank, and noise level, comparing SVP against nuclear‑norm minimization, alternating minimization, and other state‑of‑the‑art solvers. SVP consistently attains the same recovery accuracy while being 5–10× faster, and it remains stable up to 10 % additive noise. Real‑world matrix‑completion experiments on recommendation‑system datasets (e.g., MovieLens) and image in‑painting tasks demonstrate that SVP recovers high‑quality matrices with far fewer observed entries and with an order‑of‑magnitude speedup over Soft‑Impute, Alternating Least Squares, and the methods of Recht‑Fazel‑Parrilo, Candes‑Recht, and others.

In summary, the paper presents a simple, easy‑to‑implement algorithm that (i) works under strictly weaker RIP assumptions than prior guarantees, (ii) enjoys geometric convergence even in the presence of noise, and (iii) delivers substantial practical speedups on large‑scale problems. These properties make SVP a compelling alternative for any application requiring fast, provably accurate low‑rank matrix recovery.