Escaping Local Minima Provably in Non-convex Matrix Sensing: A Deterministic Framework via Simulated Lifting

Low-rank matrix sensing is a fundamental yet challenging nonconvex problem whose optimization landscape typically contains numerous spurious local minima, making it difficult for gradient-based optimizers to converge to the global optimum. Recent work has shown that over-parameterization via tensor lifting can convert such local minima into strict saddle points, an insight that also partially explains why massive scaling can improve generalization and performance in modern machine learning. Motivated by this observation, we propose a Simulated Oracle Direction (SOD) escape mechanism that simulates the landscape and escape direction of the over-parametrized space, without resorting to actually lifting the problem, since that would be computationally intractable. In essence, we designed a mathematical framework to project over-parametrized escape directions onto the original parameter space to guarantee a strict decrease of objective value from existing local minima. To the best of our knowledge, this represents the first deterministic framework that could escape spurious local minima with guarantee, especially without using random perturbations or heuristic estimates. Numerical experiments demonstrate that our framework reliably escapes local minima and facilitates convergence to global optima, while incurring minimal computational cost when compared to explicit tensor over-parameterization. We believe this framework has non-trivial implications for nonconvex optimization beyond matrix sensing, by showcasing how simulated over-parameterization can be leveraged to tame challenging optimization landscapes.

💡 Research Summary

The paper tackles a fundamental difficulty in low‑rank matrix sensing (MS): the presence of spurious local minima that trap gradient‑based methods. While recent works have shown that over‑parameterizing the problem—lifting the matrix variable to a higher‑order tensor—turns many of these undesirable minima into strict saddles, actually performing such a lift is computationally prohibitive because the number of parameters grows as (O((nr)^\ell)) for lifting order (\ell).
To reap the benefits of over‑parameterization without incurring its cost, the authors propose a deterministic escape mechanism called Simulated Oracle Direction (SOD). The key insight is that, under a Restricted Isometry Property (RIP) condition on the sensing operator, the gradient of the lifted objective in the high‑dimensional tensor space is non‑zero at any spurious stationary point of the original problem. This gradient defines an “oracle direction” that points toward a region of lower objective value.
SOD mathematically simulates this oracle direction and projects it back onto the original matrix space. The projection is non‑trivial because the oracle direction is a superposition of many tensor components rather than a single axis. The authors introduce two operators: a symmetric tensor‑power operator (\operatorname{sym}\ell(\cdot)) that preserves the structure of the original matrix, and a stack/unstack pair that converts between vectorized tensors and matrix forms. By applying these operators, they construct a new matrix (\check X) from the current spurious point (\hat X) such that (h(\check X) < h(\hat X)).
The paper provides rigorous guarantees. Theorem 3.1 shows that for lifting order (\ell=2) and RIP constant (\delta{2r}<1/5), a single SOD step always yields a strict decrease in the objective. For higher orders ((\ell>2)), direct projection may fail; the authors therefore design a Truncated Projected Gradient Descent (TPGD) scheme that performs a few gradient steps in the tensor space, truncates small components, and then projects back. They prove that the effect of TPGD can be simulated analytically, leading to a closed‑form expression for (\check X) that retains the descent guarantee.
Algorithmically, SOD proceeds as follows: (1) verify that the current point (\hat X) satisfies the first‑order optimality condition; (2) compute the smallest eigenvalue (\lambda_n) of (\nabla f(\hat X\hat X^\top)); (3) construct the oracle direction using the tensor‑gradient; (4) apply the symmetric‑tensor and stack/unstack operators to obtain (\check X); (5) resume standard gradient descent from (\check X). The additional computational overhead consists of a few matrix‑tensor products and is asymptotically the same as ordinary GD.
Empirical evaluation covers synthetic low‑rank matrices (r = 2, 5, n = 100) with varying RIP constants, as well as three real‑world applications: quantum state tomography, collaborative filtering, and power‑system state estimation. Across 1,000 random initializations per setting, SOD achieves a success rate above 90 % in escaping spurious minima, whereas random perturbation methods succeed in less than 50 % of runs. Moreover, the runtime increase relative to vanilla GD is under 20 %, while a true tensor lift would require more than 30× memory.
The contribution is twofold. First, it demonstrates that the geometric advantage of over‑parameterization can be harnessed deterministically without actually expanding the parameter space. Second, it provides a concrete, provably correct escape routine that works under standard RIP assumptions, bridging a gap between theory (which often relies on random perturbations) and practice (where computational resources are limited).
Limitations include the reliance on linear measurement models satisfying RIP; extending the analysis to non‑linear or data‑dependent sensing operators remains open. Additionally, while the paper discusses potential extensions to deep neural networks, concrete experiments in that regime are absent. Nonetheless, the SOD framework offers a promising template for designing deterministic escape mechanisms in broader non‑convex optimization problems.