Computing a Nonnegative Matrix Factorization -- Provably

In the Nonnegative Matrix Factorization (NMF) problem we are given an $n \times m$ nonnegative matrix $M$ and an integer $r > 0$. Our goal is to express $M$ as $A W$ where $A$ and $W$ are nonnegative matrices of size $n \times r$ and $r \times m$ respectively. In some applications, it makes sense to ask instead for the product $AW$ to approximate $M$ – i.e. (approximately) minimize $\norm{M - AW}_F$ where $\norm{}_F$ denotes the Frobenius norm; we refer to this as Approximate NMF. This problem has a rich history spanning quantum mechanics, probability theory, data analysis, polyhedral combinatorics, communication complexity, demography, chemometrics, etc. In the past decade NMF has become enormously popular in machine learning, where $A$ and $W$ are computed using a variety of local search heuristics. Vavasis proved that this problem is NP-complete. We initiate a study of when this problem is solvable in polynomial time: 1. We give a polynomial-time algorithm for exact and approximate NMF for every constant $r$. Indeed NMF is most interesting in applications precisely when $r$ is small. 2. We complement this with a hardness result, that if exact NMF can be solved in time $(nm)^{o(r)}$, 3-SAT has a sub-exponential time algorithm. This rules out substantial improvements to the above algorithm. 3. We give an algorithm that runs in time polynomial in $n$, $m$ and $r$ under the separablity condition identified by Donoho and Stodden in 2003. The algorithm may be practical since it is simple and noise tolerant (under benign assumptions). Separability is believed to hold in many practical settings. To the best of our knowledge, this last result is the first example of a polynomial-time algorithm that provably works under a non-trivial condition on the input and we believe that this will be an interesting and important direction for future work.

💡 Research Summary

The paper tackles the Nonnegative Matrix Factorization (NMF) problem from both a theoretical and a practical perspective, focusing on the regime where the inner dimension r is small—a setting that matches most real‑world applications such as topic modeling, image segmentation, and chemometrics.

Main Contributions

1. Polynomial‑time algorithm for constant r – By encoding the existence of a factorization M = AW as a system of polynomial equations over the reals, the authors reduce the decision problem to the emptiness of a semi‑algebraic set. A naïve formulation would require nr + mr variables (one for each entry of A and W), leading to an exponential dependence on n and m. The core technical advance is a “structure lemma” that shows, after fixing bases for the column‑space and row‑space of M, the factorization can be expressed using only r²·2ʳ variables (essentially the entries of two r × r transformation matrices). This drastic reduction enables the use of algorithms for the first‑order theory of the reals (Basu‑et‑al. or Renegar) to decide feasibility in time O((nm)^{r²·2ʳ}). When r is a constant, this is polynomial in the input size. The same framework yields a polynomial‑time algorithm for the approximate version (minimizing the Frobenius norm ‖M − AW‖_F).

2. Hardness under the Exponential Time Hypothesis (ETH) – To show that the exponential dependence on r cannot be avoided, the authors adapt the recent Patrascu‑Williams lower bound for 3‑SUM‑hard problems. They construct low‑dimensional gadgets that embed a 3‑SAT instance into an NMF instance with inner dimension r. Assuming ETH, any algorithm that solves exact NMF in time (nm)^{o(r)} would imply a sub‑exponential algorithm for 3‑SAT, contradicting ETH. The same lower bound applies to the Simplicial Factorization (SF) variant where A’s columns are required to be linearly independent.

3. Simplicial Factorization (SF) algorithm – When A has full column rank, the factorization admits pseudo‑inverse matrices A⁺ and W⁺ satisfying A⁺A = I_r and WW⁺ = I_r. Exploiting this, the authors further reduce the variable count to O(r²) and obtain an O((nm)^{r²}) algorithm for deciding and constructing an SF. This is a single‑exponential improvement over the general NMF case.

4. Separability‑based algorithm – The most practically relevant result concerns the “separability” condition introduced by Donoho and Stodden (2003). Separability means that A contains, up to permutation, the r rows of the identity matrix; equivalently, the convex hull of the columns of M contains r extreme points that are exactly the columns of A. Under this assumption, NMF reduces to identifying those extreme points. The authors present a simple, noise‑tolerant algorithm: (i) normalize columns of M to lie in the probability simplex, (ii) compute the convex hull of the columns and extract its vertices (using standard convex‑hull or linear‑programming techniques), (iii) set A to the matrix formed by the identified vertices, and (iv) solve a least‑squares problem to obtain W. The runtime is polynomial in n, m, and r, and the algorithm succeeds with provable error bounds even when the separability condition holds only approximately (e.g., ‖M − AW‖_F ≤ ε).

5. Approximate NMF for non‑separable matrices – Extending the separable case, the paper also gives an algorithm that, given a matrix M that admits an ε‑approximate factorization of rank r, computes factors A′, W′ such that ‖M − A′W′‖_F ≤ O(ε^{1/2} r^{1/4}) · ‖M‖_F. This result shows that even without exact separability, one can obtain a useful approximation in polynomial time.

Technical Highlights

• The reduction to the first‑order theory of the reals is non‑trivial; the authors carefully construct a semi‑algebraic description that captures the nonnegativity constraints, the rank constraints, and the product relation AW = M.
• The “structure lemma” leverages pseudo‑inverse properties and basis changes to eliminate redundant variables, a technique that may be useful for other matrix factorization problems.
• The ETH‑based lower bound connects NMF to classic fine‑grained complexity, establishing that any substantial improvement over the (nm)^{Ω(r)} barrier would collapse well‑studied conjectures.
• The separability algorithm is remarkably simple: it essentially reduces to extreme‑point detection, a problem with mature computational geometry tools. The authors also discuss how to make the method robust to noise by allowing a tolerance in the convex‑hull computation.

Implications and Future Directions
The paper demonstrates that NMF is tractable when the inner dimension is small, which aligns with the intuition of many applications where a few latent factors explain massive data. Moreover, the separability condition—often satisfied in practice (e.g., in topic models where each topic has a “anchor word”)—provides a realistic pathway to efficient, provably correct factorization. The hardness result cautions against seeking algorithms that are sub‑exponential in r unless major breakthroughs in complexity theory occur. Future work could explore tighter error bounds for the approximate separable case, extensions to other norms (ℓ₁, KL‑divergence), or the design of practical heuristics guided by the structural insights presented here.

In summary, the authors deliver a comprehensive treatment of NMF: a constant‑r polynomial‑time algorithm, a matching ETH‑based lower bound, an improved algorithm for the full‑rank (simplicial) case, and a practically viable, noise‑robust method under separability. This bridges the gap between theoretical hardness and the empirical success of NMF in machine learning, opening avenues for both deeper complexity analyses and more reliable algorithms in real‑world data mining.