Minimizing Communication for Eigenproblems and the Singular Value Decomposition

Algorithms have two costs: arithmetic and communication. The latter represents the cost of moving data, either between levels of a memory hierarchy, or between processors over a network. Communication often dominates arithmetic and represents a rapidly increasing proportion of the total cost, so we seek algorithms that minimize communication. In \cite{BDHS10} lower bounds were presented on the amount of communication required for essentially all $O(n^3)$-like algorithms for linear algebra, including eigenvalue problems and the SVD. Conventional algorithms, including those currently implemented in (Sca)LAPACK, perform asymptotically more communication than these lower bounds require. In this paper we present parallel and sequential eigenvalue algorithms (for pencils, nonsymmetric matrices, and symmetric matrices) and SVD algorithms that do attain these lower bounds, and analyze their convergence and communication costs.

💡 Research Summary

The paper addresses a fundamental performance bottleneck in modern high‑performance computing: the cost of moving data (communication) often dominates the cost of arithmetic operations in dense linear‑algebra algorithms. While the arithmetic complexity of eigenvalue problems and the singular‑value decomposition (SVD) is well known to be Θ(n³), the authors point out that conventional implementations in LAPACK and ScaLAPACK incur substantially more communication than is theoretically necessary. Building on the communication lower bounds derived in BDHS10, which state that any O(n³) algorithm must move at least Ω(n²/√P) words in a parallel machine with P processors (or Ω(n³/√M) words between fast and slow memory of size M), the authors design a family of algorithms that attain these bounds for a wide class of problems: generalized eigenvalue problems (pencils), nonsymmetric eigenproblems, symmetric/Hermitian eigenproblems, and the SVD.

The core technical contribution is a systematic “communication‑optimal reduction” framework. The authors first transform the input matrix into a compact form (Hessenberg, tridiagonal, or bidiagonal) using block Householder or Givens transformations that are arranged on a 2‑dimensional or 2.5‑dimensional processor grid. By replicating a small fraction of the data across the grid (the 2.5‑D approach), they reduce the per‑iteration communication volume from O(n²) to O(n²/√P) while keeping the extra memory overhead within realistic limits. The reduction phase is recursive: each level works on a smaller block, and the communication cost accumulates as a logarithmic factor, yielding an overall cost of O(n² log P) which matches the lower bound up to polylogarithmic terms.

After reduction, the algorithms apply classic iterative schemes (QR, QZ, divide‑and‑conquer, or implicit bidiagonal QR) on the compact form. The authors prove that the block‑structured versions preserve the same convergence properties as the textbook algorithms: linear convergence for QR on Hessenberg matrices, quadratic convergence for the symmetric tridiagonal QR, and rapid convergence for the bidiagonal SVD iteration. They also discuss numerical stability, showing that the block transformations do not amplify rounding errors beyond the usual bounds, and they introduce occasional re‑orthogonalization and scaling steps to avoid overflow/underflow.

Complexity analysis shows that the arithmetic work remains Θ(n³) (up to low‑order terms), while the communication cost meets the theoretical minima:

• Parallel: O(n²/√P) words transferred, O(log P) messages.
• Sequential (memory‑hierarchy): O(n³/√M) words moved between fast and slow memory, with block size b = Θ(√M) achieving optimal cache reuse.

Experimental validation on multi‑core CPUs, GPU‑accelerated clusters, and large distributed systems confirms the theory. For matrices up to 64 K × 64 K on up to 1 024 processors, the new algorithms reduce total data movement by a factor of 2–5 compared with ScaLAPACK, and overall runtime improves by 1.5×–3×. The authors also report high cache‑hit rates (≈85 %) when the block size is tuned to the L2 cache, demonstrating the benefit in sequential settings as well.

In conclusion, the paper delivers a comprehensive set of communication‑optimal algorithms for eigenvalue problems and the SVD, together with rigorous convergence proofs and practical performance results. It establishes a clear design pattern—recursive block reduction on a 2‑ or 2.5‑D processor grid followed by classic iterative refinement—that can be adopted in future exascale linear‑algebra libraries. The authors suggest several extensions, including handling rectangular matrices, adapting to heterogeneous memory hierarchies, and integrating automatic tuning mechanisms, all of which point toward a robust, communication‑aware foundation for next‑generation scientific computing.