Distributed Matrix Completion and Robust Factorization

If learning methods are to scale to the massive sizes of modern datasets, it is essential for the field of machine learning to embrace parallel and distributed computing. Inspired by the recent development of matrix factorization methods with rich theory but poor computational complexity and by the relative ease of mapping matrices onto distributed architectures, we introduce a scalable divide-and-conquer framework for noisy matrix factorization. We present a thorough theoretical analysis of this framework in which we characterize the statistical errors introduced by the “divide” step and control their magnitude in the “conquer” step, so that the overall algorithm enjoys high-probability estimation guarantees comparable to those of its base algorithm. We also present experiments in collaborative filtering and video background modeling that demonstrate the near-linear to superlinear speed-ups attainable with this approach.

💡 Research Summary

The paper addresses the scalability bottleneck of high‑accuracy matrix factorization (MF) methods—particularly noisy matrix completion and robust matrix factorization—by introducing a divide‑factor‑combine (DFC) framework that enables parallel and distributed execution without sacrificing statistical guarantees. The authors first formalize the problem: an observed matrix (M = L_0 + S_0 + Z_0) where (L_0) is low‑rank, (S_0) is a sparse outlier matrix, and (Z_0) is dense noise. Two settings are considered: (1) noisy matrix completion (MC) where only a random subset of entries is observed and there are no outliers, and (2) robust matrix factorization (RMF) where all entries are observed but a few are arbitrarily corrupted.

The DFC framework consists of three steps. In the Divide step the observed matrix (P_\Omega(M)) is randomly partitioned into (t) submatrices. Three variants are described: (i) column‑wise partition (used by DFC‑PROJ and DFC‑RP), (ii) a combination of a random column block and a random row block (used by DFC‑NYS). In the Factor step each submatrix is processed independently by any “base” MF algorithm—typically a nuclear‑norm regularized convex optimizer that enjoys strong theoretical error bounds but is computationally heavy because it repeatedly computes truncated SVDs. Because the base algorithm returns low‑rank factors, the per‑subproblem cost drops dramatically.

The Combine step merges the sub‑solutions using one of three randomized low‑rank approximation techniques:

• Column Projection (DFC‑PROJ) projects all sub‑estimates onto the column space of the first sub‑estimate, following the column‑sampling theory of Frieze et al. This yields a global low‑rank matrix ( \widehat L_{\text{proj}} ) with error that scales with the number of sampled columns (l).

• Random Projection (DFC‑RP) builds a common low‑dimensional subspace via a Gaussian matrix (G) and a few power‑iteration steps, then approximates the full matrix as ( QQ^{+}M ). The Johnson‑Lindenstrauss lemma and Halko et al.’s analysis guarantee that, for target rank (k) and modest oversampling (p), the approximation error is bounded by a small multiple of the optimal error.

• Generalized Nyström (DFC‑NYS) samples a set of columns (C) and rows (R), forms the cross‑matrix (W), and reconstructs ( \widehat L_{\text{nys}} = C W^{+} R ). This method extends the classic Nyström technique to arbitrary (non‑symmetric) matrices and retains the same error guarantees as column sampling when (l) and (d) are sufficiently large.

The authors provide a rigorous statistical analysis. First, they show that each sub‑problem inherits the same high‑probability error bound as the original MF algorithm, because the random partition preserves the underlying low‑rank structure with high probability. Second, they bound the additional error introduced by the combine step using spectral properties of random sampling and projection. The main theorem states that, provided the column (or row) sample sizes satisfy ( l, d = \Omega(k \log k) ), the overall estimator ( \widehat L ) satisfies \