Spectral approximations in machine learning

In many areas of machine learning, it becomes necessary to find the eigenvector decompositions of large matrices. We discuss two methods for reducing the computational burden of spectral decompositions: the more venerable Nystom extension and a newly introduced algorithm based on random projections. Previous work has centered on the ability to reconstruct the original matrix. We argue that a more interesting and relevant comparison is their relative performance in clustering and classification tasks using the approximate eigenvectors as features. We demonstrate that performance is task specific and depends on the rank of the approximation.

💡 Research Summary

The paper addresses a practical problem that arises in many modern machine learning pipelines: the need to compute eigen‑decompositions of very large, positive‑definite kernel matrices (typically graph Laplacians) in order to obtain low‑dimensional spectral embeddings such as diffusion maps. Exact eigendecomposition scales as O(n³) and quickly becomes infeasible for datasets with thousands or millions of points. The authors therefore compare two well‑known approximation strategies that reduce the computational burden while still delivering usable eigenvectors for downstream learning tasks.

Methods compared

1. Nyström extension – A classical technique that samples a subset of m columns (or rows) of the kernel matrix, forms the small m×m sub‑matrix W(m), computes its eigen‑decomposition, and then extends the resulting eigenvectors to the full n‑dimensional space using a simple analytic formula. The paper discusses two sampling schemes: uniform random sampling (the most common) and a “weighted” version that draws columns with probability proportional to the diagonal entries of the kernel.

2. Gaussian projection – A more recent algorithm (cited as