Managing Communication Latency-Hiding at Runtime for Parallel Programming Languages and Libraries

This work introduces a runtime model for managing communication with support for latency-hiding. The model enables non-computer science researchers to exploit communication latency-hiding techniques seamlessly. For compiled languages, it is often possible to create efficient schedules for communication, but this is not the case for interpreted languages. By maintaining data dependencies between scheduled operations, it is possible to aggressively initiate communication and lazily evaluate tasks to allow maximal time for the communication to finish before entering a wait state. We implement a heuristic of this model in DistNumPy, an auto-parallelizing version of numerical Python that allows sequential NumPy programs to run on distributed memory architectures. Furthermore, we present performance comparisons for eight benchmarks with and without automatic latency-hiding. The results shows that our model reduces the time spent on waiting for communication as much as 27 times, from a maximum of 54% to only 2% of the total execution time, in a stencil application.

💡 Research Summary

The paper presents a runtime model that automatically hides communication latency for parallel programming languages and libraries, with a focus on interpreted languages such as Python. Traditional latency‑hiding techniques rely on compile‑time analysis to generate efficient communication‑computation schedules. This approach works well for compiled languages (e.g., C/C++ with MPI) but is largely infeasible for dynamic, interpreted environments where code and data structures are created at runtime.

To bridge this gap, the authors introduce a data‑dependency graph that is built dynamically as the program executes. Each operation (read, write, or compute) becomes a node in the graph, and edges represent true data dependencies. When an operation requires a remote data block that is not yet locally available, the runtime immediately initiates a non‑blocking MPI communication (MPI_Isend/MPI_Irecv) and marks the operation as “pending.” The actual computation is postponed until the pending communication completes. In the meantime, the scheduler scans the graph for other operations whose dependencies are already satisfied and executes them, thereby keeping the CPU busy while the network transfers data. This “early‑communication, lazy‑evaluation” strategy maximizes the overlap between communication and computation without any programmer intervention.

The model is implemented as a heuristic scheduler inside DistNumPy, an auto‑parallelizing extension of NumPy that automatically distributes array operations across a cluster. The scheduler classifies each NumPy operation, determines the owning process for the required array slices, and issues asynchronous transfers as soon as a remote slice is needed. Completion of transfers is checked with MPI_Test; if a pending operation cannot proceed, the scheduler looks for independent work. The approach is fully automatic: users write ordinary sequential NumPy code, and DistNumPy transparently applies the latency‑hiding runtime.

Evaluation is performed on eight representative scientific kernels, including 2‑D and 3‑D stencil updates, dense matrix multiplication, FFT, LU decomposition, and others. Experiments run on a 64‑core InfiniBand cluster compare two configurations: (1) DistNumPy without latency‑hiding (baseline) and (2) DistNumPy with the proposed runtime model. Results show dramatic reductions in time spent waiting for communication. In the stencil benchmark, waiting time drops from 54 % of total execution to just 2 %, a 27‑fold improvement. Overall execution time improves by an average of 1.8×, with the most pronounced gains (up to 2.5×) on communication‑intensive workloads. Overhead analysis indicates that graph construction and dependency checks consume less than 3 % of total runtime, even for moderate problem sizes. Scaling tests demonstrate that the benefit persists as the number of processes increases; the scheduler continues to find enough independent work to hide network latency effectively.

The authors acknowledge several limitations. First, the dynamic graph incurs a modest cost that can dominate very small problem instances. Second, the current implementation targets only CPU‑MPI clusters; extending the model to GPUs, RDMA, or other heterogeneous accelerators would require additional mechanisms for tracking device‑side dependencies. Third, the heuristic scheduler does not guarantee optimal schedules for highly irregular or non‑linear computation patterns; in such cases sub‑optimal overlap may occur. Future work is outlined to address these issues: (a) hybrid static‑dynamic analysis to reduce graph‑building overhead, (b) multi‑level scheduling that incorporates heterogeneous memory hierarchies, and (c) machine‑learning‑guided heuristics that predict the best ordering of pending operations.

In conclusion, the paper demonstrates that automatic latency‑hiding is feasible for interpreted, high‑level scientific languages. By maintaining a runtime dependency graph and employing lazy evaluation, the system can aggressively pre‑fetch remote data and keep the processor busy, achieving performance gains comparable to hand‑tuned MPI code. The integration with DistNumPy shows that existing Python codebases can be scaled to distributed memory systems with minimal programmer effort, thereby broadening the accessibility of high‑performance computing to a wider scientific community.