Uniform Sampling for Matrix Approximation

Random sampling has become a critical tool in solving massive matrix problems. For linear regression, a small, manageable set of data rows can be randomly selected to approximate a tall, skinny data matrix, improving processing time significantly. For theoretical performance guarantees, each row must be sampled with probability proportional to its statistical leverage score. Unfortunately, leverage scores are difficult to compute. A simple alternative is to sample rows uniformly at random. While this often works, uniform sampling will eliminate critical row information for many natural instances. We take a fresh look at uniform sampling by examining what information it does preserve. Specifically, we show that uniform sampling yields a matrix that, in some sense, well approximates a large fraction of the original. While this weak form of approximation is not enough for solving linear regression directly, it is enough to compute a better approximation. This observation leads to simple iterative row sampling algorithms for matrix approximation that run in input-sparsity time and preserve row structure and sparsity at all intermediate steps. In addition to an improved understanding of uniform sampling, our main proof introduces a structural result of independent interest: we show that every matrix can be made to have low coherence by reweighting a small subset of its rows.

💡 Research Summary

The paper addresses a fundamental problem in large‑scale linear regression and related matrix computations: how to reduce a tall‑skinny data matrix A ∈ ℝⁿˣᵈ by sampling a small subset of its rows while preserving the spectral properties needed for accurate regression solutions. Classical results guarantee that sampling rows with probabilities proportional to their statistical leverage scores τᵢ = aᵢᵀ(AᵀA)⁺aᵢ yields a λ‑spectral approximation of A with only O(d log d) rows. However, computing exact leverage scores requires forming (AᵀA)⁺, which is as expensive as solving the original regression problem.

The authors revisit the simplest alternative—uniform random row sampling—and ask what useful information uniform sampling actually retains. Their main technical contribution, Theorem 1, shows that if one samples O(m) rows uniformly to form a sketch \tilde A, then the leverage‑score estimates \tilde τᵢ derived from \tilde A satisfy two crucial properties: (i) they are always upper bounds on the true scores ( \tilde τᵢ ≥ τᵢ for all i ), and (ii) the expected total sum of the estimates is bounded by n·d/m. Consequently, by choosing m = Θ(d log d) one can guarantee that Σ_i \tilde τᵢ ≤ O(d), which is the same bound that holds for exact leverage scores. This result eliminates the need for per‑row multiplicative approximations and shows that a weak, aggregate guarantee suffices for the next sampling round.

A second, structural insight is Lemma 1 (Coherence‑Reducing Reweighting). For any matrix A and any target coherence level α > 0, there exists a diagonal weighting matrix W with at most d/α entries strictly less than 1 such that every row of WA has leverage score at most α. The lemma is existential; the algorithm never needs to construct W explicitly. Its significance lies in the fact that once a matrix has low coherence, uniform sampling alone provides a full spectral approximation. By combining Lemma 1 with Theorem 1, the authors argue that even if A is highly coherent, a uniformly sampled sketch can be used to estimate leverage scores that are sufficiently small to drive the iterative process forward.

Building on these two results, the paper proposes a simple iterative row‑sampling framework that runs in input‑sparsity time (O(nnz(A))) and preserves both row structure and sparsity throughout. The high‑level loop is:

1. Uniform reduction – Sample O(m) rows uniformly from A to obtain a sketch \tilde A.
2. Leverage‑score estimation – Compute an approximation of ( \tilde Aᵀ\tilde A )⁺ (e.g., via a few power‑iterations or a fast sketch) and form upper‑bound estimates \tilde τᵢ for all rows of A.
3. Importance resampling – Sample rows of A with probabilities proportional to \tilde τᵢ, thereby dramatically shrinking the matrix size while guaranteeing a λ‑spectral approximation.
4. Recursion – Repeat the process on the newly obtained matrix.

Because each iteration reduces the number of rows by a constant factor (e.g., halves it) while keeping Σ_i \tilde τᵢ = O(d), after O(log n) iterations the matrix size drops to O(d log d) rows, achieving a λ‑spectral approximation with λ = O(1). Importantly, no dense Johnson‑Lindenstrauss transforms are required; the algorithm never mixes rows, so the intermediate matrices remain sparse and retain the original row semantics (crucial for graph incidence matrices, feature‑wise data, etc.).

The authors also present alternative versions of Theorem 1 (Theorems 3 and 4) that use slightly different estimation techniques, further illustrating the flexibility of the approach. They discuss how the coherence‑reducing reweighting can be viewed as a theoretical tool that justifies the uniform‑sampling step without incurring any computational overhead.

Overall, the paper delivers three key messages:

• Uniform row sampling, previously thought too weak for general matrices, actually provides enough aggregate leverage‑score information to drive an iterative refinement process.
• A small set of reweighted rows can always reduce a matrix’s coherence, making uniform sampling effective on the reweighted matrix and, by extension, on the original matrix via the bound arguments.
• The resulting iterative algorithm matches the best known runtimes for spectral approximation (input‑sparsity time) while preserving sparsity and row structure, unlike prior methods that rely on dense random projections or require many passes over the data.

These contributions deepen the theoretical understanding of uniform sampling in randomized numerical linear algebra and open the door to practical, structure‑preserving matrix sketching techniques for massive data sets.