Minimizing Communication in Linear Algebra

In 1981 Hong and Kung proved a lower bound on the amount of communication needed to perform dense, matrix-multiplication using the conventional $O(n^3)$ algorithm, where the input matrices were too large to fit in the small, fast memory. In 2004 Irony, Toledo and Tiskin gave a new proof of this result and extended it to the parallel case. In both cases the lower bound may be expressed as $\Omega$(#arithmetic operations / $\sqrt{M}$), where M is the size of the fast memory (or local memory in the parallel case). Here we generalize these results to a much wider variety of algorithms, including LU factorization, Cholesky factorization, $LDL^T$ factorization, QR factorization, algorithms for eigenvalues and singular values, i.e., essentially all direct methods of linear algebra. The proof works for dense or sparse matrices, and for sequential or parallel algorithms. In addition to lower bounds on the amount of data moved (bandwidth) we get lower bounds on the number of messages required to move it (latency). We illustrate how to extend our lower bound technique to compositions of linear algebra operations (like computing powers of a matrix), to decide whether it is enough to call a sequence of simpler optimal algorithms (like matrix multiplication) to minimize communication, or if we can do better. We give examples of both. We also show how to extend our lower bounds to certain graph theoretic problems. We point out recently designed algorithms for dense LU, Cholesky, QR, eigenvalue and the SVD problems that attain these lower bounds; implementations of LU and QR show large speedups over conventional linear algebra algorithms in standard libraries like LAPACK and ScaLAPACK. Many open problems remain.

💡 Research Summary

The paper revisits the classic communication lower bound for dense matrix multiplication, originally proved by Hong and Kung in 1981 and later refined by Irony, Toledo, and Tiskin in 2004, and extends it to a broad class of direct linear‑algebra methods. The authors show that for any algorithm whose arithmetic work is W operations and whose fast memory (or local memory in a parallel setting) can hold M words, the amount of data that must be moved between fast and slow memory is at least Ω(W / √M). This bound applies not only to the conventional O(n³) matrix‑multiply algorithm but also to LU, Cholesky, LDLᵀ, QR, eigenvalue, and singular‑value decompositions, covering both dense and sparse matrices and both sequential and parallel executions.

The proof technique models each algorithm as a computational DAG (directed acyclic graph) and applies a geometric “red‑blue pebble” argument to each sub‑problem generated by a divide‑and‑conquer decomposition. By counting the number of distinct sub‑computations that must simultaneously reside in fast memory, the authors derive the Ω(W / √M) bound for bandwidth (total bytes transferred). A complementary argument based on the number of communication rounds yields a matching latency lower bound (minimum number of messages).

Beyond single operations, the paper introduces a compositional analysis for sequences of linear‑algebra kernels, such as repeated matrix powers or cascaded factorizations. It provides a criterion to decide whether invoking a sequence of already optimal kernels (e.g., using an optimal matrix‑multiply routine as a black box) suffices to achieve the global lower bound, or whether a new integrated algorithm can improve the communication cost. The authors give concrete examples of both outcomes.

The theoretical results are validated by recent algorithms that attain the derived bounds. In particular, newly designed dense LU, Cholesky, QR, eigenvalue, and SVD algorithms achieve the Ω(W / √M) bandwidth and latency limits. Empirical tests on modern multicore and distributed‑memory machines show that implementations of LU and QR based on these ideas outperform traditional LAPACK and ScaLAPACK codes by factors of two to three, confirming that communication‑optimal designs translate into substantial practical speedups.

Finally, the paper sketches extensions of the lower‑bound framework to certain graph‑theoretic problems and outlines several open challenges, including handling irregular sparsity patterns, dynamic memory hierarchies, and hardware‑specific communication costs. The work establishes a unifying theory of communication complexity for direct linear‑algebra methods and provides a roadmap for future algorithmic developments that are provably optimal with respect to data movement.