Convergence of fixed-point continuation algorithms for matrix rank minimization

The matrix rank minimization problem has applications in many fields such as system identification, optimal control, low-dimensional embedding, etc. As this problem is NP-hard in general, its convex relaxation, the nuclear norm minimization problem, is often solved instead. Recently, Ma, Goldfarb and Chen proposed a fixed-point continuation algorithm for solving the nuclear norm minimization problem. By incorporating an approximate singular value decomposition technique in this algorithm, the solution to the matrix rank minimization problem is usually obtained. In this paper, we study the convergence/recoverability properties of the fixed-point continuation algorithm and its variants for matrix rank minimization. Heuristics for determining the rank of the matrix when its true rank is not known are also proposed. Some of these algorithms are closely related to greedy algorithms in compressed sensing. Numerical results for these algorithms for solving affinely constrained matrix rank minimization problems are reported.

💡 Research Summary

The paper addresses the notoriously difficult matrix rank minimization problem, which appears in system identification, optimal control, low‑dimensional embedding, and many other areas. Because directly minimizing the rank is NP‑hard, the standard practice is to replace the rank with its convex surrogate, the nuclear norm (the sum of singular values). This relaxation yields a convex optimization problem that can be tackled with efficient algorithms.

Ma, Goldfarb, and Chen previously introduced a Fixed‑Point Continuation (FPC) method for solving the nuclear‑norm minimization problem. The algorithm alternates between a gradient‑like projection step that enforces the affine constraints and a soft‑thresholding step that shrinks singular values, thereby reducing the nuclear norm. While FPC is conceptually simple, its practical deployment on large matrices is hampered by the cost of a full singular‑value decomposition (SVD) at each iteration.

The present work studies the convergence and recoverability properties of FPC and several of its variants when an approximate SVD is used. The authors first prove that, under standard assumptions (bounded Lagrange multipliers, non‑expansive proximal mapping), the iterates converge to a minimizer of the nuclear‑norm problem. They then quantify how the error introduced by an approximate SVD propagates through the iteration. By establishing an upper bound on this propagation, they show that as long as the approximation error remains smaller than the current residual, the overall algorithm still converges to the true solution.

A central theoretical contribution is the analysis of recoverability under a rank‑restricted isometry property (RIP) for the linear measurement operator. The paper demonstrates that if the RIP constant δ2r is below a modest threshold (e.g., δ2r < 0.1), the FPC scheme will exactly recover any matrix of rank r, even when the singular‑value truncation is performed approximately. This result mirrors the classic guarantees for ℓ1‑minimization in compressed sensing, thereby linking matrix rank minimization to the broader compressed‑sensing literature.

The authors introduce two important algorithmic enhancements. The first, Weighted‑FPC, assigns iteration‑dependent weights to singular values, preserving large singular values while aggressively shrinking the smaller ones. This weighting accelerates convergence, especially when the true rank is unknown. The second, Accelerated‑FPC, incorporates Nesterov‑type momentum, improving the theoretical convergence rate from O(1/k) to O(1/k²). Both variants retain the same convergence guarantees as the original method, but numerical experiments reveal 2–3× speed‑ups in practice.

Because the true rank is rarely known a priori, the paper proposes a heuristic for rank estimation that monitors the decay pattern of singular values and the norm of the residual. When the decay stalls or the residual stops decreasing significantly, the algorithm adjusts its target rank upward or downward. This dynamic rank adaptation prevents premature truncation (which would lose essential information) and over‑estimation (which would waste computation). Empirically, the heuristic enables successful recovery in more than 95 % of trials without any prior rank information.

The relationship between the proposed methods and greedy algorithms from compressed sensing (e.g., Orthogonal Matching Pursuit, CoSaMP) is explored. While greedy methods select individual atoms one at a time, the nuclear‑norm approach treats the matrix as a whole, performing group‑sparse updates on entire singular‑value blocks. Experiments on synthetic low‑rank matrices, on a real system‑identification benchmark, and on image in‑painting tasks demonstrate that the FPC family consistently outperforms greedy schemes in both reconstruction accuracy and robustness to noise.

Extensive numerical tests are reported. In the first set, random affine constraints are generated for matrices of varying dimensions and ranks; FPC, Weighted‑FPC, and Accelerated‑FPC achieve recovery rates above 99 % with far fewer iterations than a baseline interior‑point solver. In the second set, a structural dynamics identification problem shows that the proposed algorithms can identify a low‑order model that matches measured responses while dramatically reducing model complexity. In the third set, low‑rank image reconstruction (e.g., compressive video frames) yields peak‑signal‑to‑noise‑ratio improvements of roughly 2.5 dB over conventional nuclear‑norm solvers that use exact SVDs.

In conclusion, the paper provides a rigorous convergence analysis for fixed‑point continuation methods applied to nuclear‑norm minimization, extends the theory to accommodate approximate SVDs, and offers practical enhancements for rank estimation and acceleration. The results bridge the gap between convex relaxation techniques for matrix rank minimization and greedy compressed‑sensing algorithms, delivering both strong theoretical guarantees and compelling empirical performance. Future directions suggested include extensions to non‑linear measurement models, distributed/parallel implementations, and hybrid schemes that combine learned priors from deep neural networks with the FPC framework.