Fast and Faster: A Comparison of Two Streamed Matrix Decomposition Algorithms

With the explosion of the size of digital dataset, the limiting factor for decomposition algorithms is the \emph{number of passes} over the input, as the input is often stored out-of-core or even off-site. Moreover, we’re only interested in algorithms that operate in \emph{constant memory} w.r.t. to the input size, so that arbitrarily large input can be processed. In this paper, we present a practical comparison of two such algorithms: a distributed method that operates in a single pass over the input vs. a streamed two-pass stochastic algorithm. The experiments track the effect of distributed computing, oversampling and memory trade-offs on the accuracy and performance of the two algorithms. To ensure meaningful results, we choose the input to be a real dataset, namely the whole of the English Wikipedia, in the application settings of Latent Semantic Analysis.

💡 Research Summary

The paper “Fast and Faster: A Comparison of Two Streamed Matrix Decomposition Algorithms” presents a thorough empirical study of three streaming matrix‑decomposition methods applied to a truly massive real‑world dataset: the entire English Wikipedia (≈3.2 million documents, 100 000 vocabulary terms, 0.5 billion non‑zero entries). The authors focus on two fundamental constraints that dominate modern large‑scale analytics: (1) the number of passes over the input data, because data are often stored out‑of‑core or remotely, and (2) the memory footprint, which must remain constant with respect to the number of observations (columns) so that arbitrarily large streams can be processed.

The three algorithms examined are:

• P1 – One‑Pass Distributed Algorithm (Radim Řehůrek, 2010). The input stream is split into document chunks that are processed independently on a cluster. Each chunk undergoes an in‑core singular‑value decomposition (using SVDLIBC or a similar black‑box). The partial decompositions are then merged in a communication‑light fashion. Memory usage is O(m·(k + l)) and independent of n, the number of columns. The method is truly one‑pass, making it attractive for streaming environments where data cannot be revisited.

• P2 – Two‑Pass Stochastic Algorithm (Halko et al., 2009). This method builds a random sketch Y = AΩ in a single pass, optionally refines it with q power iterations (each requiring an extra pass), orthogonalizes Y to obtain Q, and finally computes a small (k + l) × (k + l) covariance matrix X = (QᵀA)(QᵀA)ᵀ. The eigen‑decomposition of X yields the truncated singular vectors of A. The algorithm never materializes any O(n) or O(m²) matrix, keeping memory at O(m·(k + l)). Accuracy depends heavily on the oversampling factor l and the number of power iterations q.

• P12 – Hybrid Algorithm. The authors replace the in‑core SVD used in P1’s merging step with the stochastic decomposition of P2, thereby retaining the one‑pass communication pattern while gaining the higher accuracy of the random‑projection approach.

All experiments fix the target rank k = 400 and use three 2.0 GHz Intel Xeon workstations (4 GB RAM each) connected via Ethernet. Because the machines were shared, each configuration was run twice and the better run reported.

Key experimental dimensions

1. Oversampling (l) and Power Iterations (q).

   • For P2, increasing l from 0 to 400 and q from 0 to 3 dramatically improves the recovered singular values. With l = 400 and q = 3, P2’s spectrum matches that of the “ground‑truth” (approximated by a highly oversampled run) and outperforms P1/P12.
   • P1 and P12 also benefit from oversampling, but the effect is milder; l = 200 already yields near‑optimal accuracy.
   • Runtime grows with l for all methods; P2’s runtime additionally scales with q because each power iteration adds a full pass over the data.

2. Chunk Size.

   • Chunk sizes of 10 k, 20 k, and 40 k documents were tested for P1 and P12. Larger chunks reduce the number of merge operations, yielding a modest speed‑up (≈30 % from 10 k to 40 k) while leaving accuracy essentially unchanged. This confirms that the merge step, not the local decomposition, dominates the cost.

3. Input Order Sensitivity.

   • The Wikipedia stream is naturally ordered alphabetically. When the authors shuffled the stream before feeding it to P1, the resulting singular values shifted noticeably, indicating that a single‑pass algorithm is sensitive to “subspace drift” caused by non‑random observation order. P2, by virtue of multiple passes, is invariant to ordering.

4. Distributed Scaling.

   • P1 and P12 scale almost linearly with the number of nodes: 1 node ≈ 10 h 36 m, 2 nodes ≈ 6 h 0 m, 4 nodes ≈ 3 h 18 m. Communication overhead is negligible because only the initial chunk distribution and final result collection occur. P2 could be distributed only if the data were already placed on a distributed filesystem; otherwise the extra passes make network traffic the dominant bottleneck, so the authors omitted distributed P2 results.

5. Qualitative Topic Extraction.

   • Using P2 with three power iterations and l = 400, the top ten latent topics correspond to high‑level Wikipedia meta‑categories (administration, country lists, film, sports, music, etc.), demonstrating that the algorithm captures meaningful semantic structure at scale.

Conclusions and Practical Implications

• When the primary constraint is the number of passes (e.g., streaming pipelines, remote storage), the one‑pass methods P1 and its hybrid P12 are the only viable options. Their linear scalability on commodity clusters and modest memory footprint make them attractive for production environments.

• If accuracy is paramount and the data can be read multiple times (or already resides on a distributed file system), the two‑pass stochastic algorithm P2 is superior, provided sufficient oversampling (l ≈ k + 100) and a few power iterations (q ≥ 2) are employed.

• The sensitivity of P1 to input ordering suggests that practitioners should either randomize the stream or adopt larger chunks to mitigate drift. Future work could explore adaptive re‑orthogonalization or incremental subspace correction within the one‑pass framework.

• The authors released a Python implementation as part of the open‑source gensim library. By leveraging NumPy’s BLAS bindings, the code achieves competitive performance despite being written in a high‑level language, facilitating reproducibility and easy integration into NLP pipelines.

Overall, the paper provides a clear, data‑driven roadmap for choosing between single‑pass distributed SVD and multi‑pass stochastic sketching in large‑scale latent semantic analysis, balancing the trade‑offs among I/O cost, memory usage, parallel scalability, and decomposition accuracy.