Randomized algorithms for matrices and data

Randomized algorithms for very large matrix problems have received a great deal of attention in recent years. Much of this work was motivated by problems in large-scale data analysis, and this work was performed by individuals from many different research communities. This monograph will provide a detailed overview of recent work on the theory of randomized matrix algorithms as well as the application of those ideas to the solution of practical problems in large-scale data analysis. An emphasis will be placed on a few simple core ideas that underlie not only recent theoretical advances but also the usefulness of these tools in large-scale data applications. Crucial in this context is the connection with the concept of statistical leverage. This concept has long been used in statistical regression diagnostics to identify outliers; and it has recently proved crucial in the development of improved worst-case matrix algorithms that are also amenable to high-quality numerical implementation and that are useful to domain scientists. Randomized methods solve problems such as the linear least-squares problem and the low-rank matrix approximation problem by constructing and operating on a randomized sketch of the input matrix. Depending on the specifics of the situation, when compared with the best previously-existing deterministic algorithms, the resulting randomized algorithms have worst-case running time that is asymptotically faster; their numerical implementations are faster in terms of clock-time; or they can be implemented in parallel computing environments where existing numerical algorithms fail to run at all. Numerous examples illustrating these observations will be described in detail.

💡 Research Summary

**
This monograph surveys recent advances in randomized algorithms for two fundamental matrix problems that arise ubiquitously in large‑scale data analysis: over‑determined linear least‑squares (LS) and low‑rank matrix approximation. The authors focus on two families of techniques—random sampling and random projection—and on a unifying concept, statistical leverage, that guides the design of importance‑sampling distributions and informs the analysis of projection methods.

Random sampling algorithms select a small subset of rows (for LS) or columns/rows (for CUR‑type low‑rank approximations) with probabilities proportional to the leverage scores, i.e., the diagonal entries of the orthogonal projector onto the top‑k singular subspace of the input matrix. After appropriate rescaling, the sampled submatrix forms a “sketch” on which a standard deterministic solver is applied. The resulting solution enjoys relative‑error guarantees of the form
‖A x̃ − b‖₂ ≤ (1+ε)‖A x_opt − b‖₂ for LS, and
‖A − C U R‖_F ≤ (1+ε)‖A − A_k‖_F for CUR,
with high probability. The number of samples needed scales only with k log k/ε², independent of the ambient dimensions, yielding asymptotic speed‑ups over classical O(m n²) algorithms.

Random projection algorithms replace the original matrix A by a sketch S = A Ω, where Ω is a suitably chosen random matrix (e.g., sub‑Gaussian, SRHT, CountSketch). The sketch preserves the column space of A up to (1±ε) distortion, enabling a fast computation of an orthonormal basis Q for the range of A. The low‑rank approximation A ≈ QQᵀA satisfies an absolute‑error bound ‖A − QQᵀA‖₂ ≤ ε‖A‖₂, and the LS problem can be solved on the much smaller system S x ≈ b′, leading to O(m n log k) or O(m n ε⁻²) runtimes for dense matrices.

A central technical contribution is the decoupling of randomization from linear algebra. By first constructing a fast, coarse sketch (often via a random projection) and then using it to estimate leverage scores, one obtains accurate importance‑sampling probabilities without ever forming the full SVD. This two‑stage “hybrid” approach yields overall running times of O(nnz(A) log n + poly(k,ε⁻¹)), which are near‑optimal for sparse data.

The monograph also discusses practical implementation issues. Computing exact leverage scores is expensive; the authors propose efficient approximations using a preliminary projection, followed by a small SVD on the projected matrix. They describe how to embed these steps in modern parallel and distributed frameworks such as MapReduce, Spark, and GPU‑accelerated libraries, emphasizing that the sketching operations are embarrassingly parallel and communication‑light.

Empirical sections illustrate the theory on real datasets from genomics, social networks, and image processing. In each case, leverage scores identify influential rows or columns (e.g., high‑degree nodes, outlier genes, salient image patches). Sampling based on these scores dramatically reduces the data size while preserving predictive accuracy, and projection‑based methods achieve comparable or better accuracy with substantially lower wall‑clock time than LAPACK/ARPACK baselines.

Finally, the authors reflect on lessons learned: (1) statistical leverage provides a principled bridge between worst‑case theoretical guarantees and domain‑specific interpretability; (2) randomization, when combined with problem‑specific structure, yields algorithms that are both fast and robust; (3) the hybrid sampling‑projection pipeline is a versatile template for future extensions to streaming, online, and non‑linear settings such as kernel methods and deep learning. The paper concludes by outlining open research directions, including tighter bounds for adaptive sampling, sketching for tensor problems, and integration with modern machine‑learning pipelines.