A Class of Parallel Tiled Linear Algebra Algorithms for Multicore Architectures

As multicore systems continue to gain ground in the High Performance Computing world, linear algebra algorithms have to be reformulated or new algorithms have to be developed in order to take advantage of the architectural features on these new processors. Fine grain parallelism becomes a major requirement and introduces the necessity of loose synchronization in the parallel execution of an operation. This paper presents an algorithm for the Cholesky, LU and QR factorization where the operations can be represented as a sequence of small tasks that operate on square blocks of data. These tasks can be dynamically scheduled for execution based on the dependencies among them and on the availability of computational resources. This may result in an out of order execution of the tasks which will completely hide the presence of intrinsically sequential tasks in the factorization. Performance comparisons are presented with the LAPACK algorithms where parallelism can only be exploited at the level of the BLAS operations and vendor implementations.

💡 Research Summary

The paper addresses the growing mismatch between traditional dense linear‑algebra libraries (primarily LAPACK) and the architectural realities of modern multicore processors. LAPACK’s parallelism is confined to the BLAS level‑3 calls, which means that synchronization points and memory‑bandwidth contention become severe bottlenecks as the number of cores increases. To overcome these limitations, the authors propose a tiled formulation of the three fundamental factorizations—Cholesky, LU, and QR—where the matrix is partitioned into small, fixed‑size square blocks (tiles). Each tile is processed by a fine‑grained task that corresponds to a BLAS‑3 kernel such as POTRF, TRSM, SYRK, or GEMM.

The central contribution is the explicit construction of a directed acyclic graph (DAG) that captures data dependencies among tasks. A lightweight runtime scheduler dynamically maintains a “ready list” of tasks whose predecessors have completed. When a core becomes idle, the scheduler selects a ready task, respecting a simple priority rule (tasks with fewer remaining dependencies are favored). As tasks finish, dependency counters are atomically decremented, instantly exposing new ready tasks. This dynamic, out‑of‑order execution hides the inherently sequential phases of the factorization (e.g., panel factorizations) and allows the computation to keep all cores busy.

Tile size is a critical tuning parameter. Small tiles increase the number of parallel tasks but reduce per‑task computational intensity, leading to lower cache reuse. Large tiles improve arithmetic intensity but limit concurrency. Empirical results on Intel Xeon platforms indicate that 64×64 or 128×128 tiles strike a good balance for the problem sizes examined (from a few thousand to ten thousand dimensions).

Implementation details include contiguous memory layout for tiles to maximize cache locality, lock‑free queues for the ready list, and atomic operations for dependency management. The authors also employ a look‑ahead technique during DAG construction to eliminate unnecessary dependencies, thereby deepening the execution pipeline and improving load balance.

Performance experiments compare the tiled, dynamically scheduled algorithms against the best‑available LAPACK‑BLAS implementations on 8‑core and 12‑core Xeon systems. For a 4 000 × 4 000 Cholesky factorization, the tiled approach achieves roughly 3.5 GFLOPS versus 1.8 GFLOPS for the traditional code, representing a speed‑up of nearly 2×. Similar gains are observed for LU and QR factorizations, with speed‑ups ranging from 1.8× to 3.5× depending on matrix size and core count. The scheduler maintains high core utilization (often above 90 %) even when the workload is irregular, demonstrating robustness to load imbalance. Memory usage remains comparable to the conventional approach, while cache‑miss rates drop significantly due to the tile‑centric data access pattern.

The authors conclude that the DAG‑driven, tile‑based methodology provides a scalable foundation for dense linear algebra on multicore CPUs and can be extended to heterogeneous environments such as GPU‑accelerated systems or CPU‑GPU hybrids. By decoupling algorithmic structure from static parallel loops and embracing fine‑grained, dependency‑aware scheduling, the work paves the way for next‑generation high‑performance linear‑algebra libraries that fully exploit the parallel potential of emerging hardware.