Many-Task Computing and Blue Waters

This report discusses many-task computing (MTC) generically and in the context of the proposed Blue Waters systems, which is planned to be the largest NSF-funded supercomputer when it begins production use in 2012. The aim of this report is to inform the BW project about MTC, including understanding aspects of MTC applications that can be used to characterize the domain and understanding the implications of these aspects to middleware and policies. Many MTC applications do not neatly fit the stereotypes of high-performance computing (HPC) or high-throughput computing (HTC) applications. Like HTC applications, by definition MTC applications are structured as graphs of discrete tasks, with explicit input and output dependencies forming the graph edges. However, MTC applications have significant features that distinguish them from typical HTC applications. In particular, different engineering constraints for hardware and software must be met in order to support these applications. HTC applications have traditionally run on platforms such as grids and clusters, through either workflow systems or parallel programming systems. MTC applications, in contrast, will often demand a short time to solution, may be communication intensive or data intensive, and may comprise very short tasks. Therefore, hardware and software for MTC must be engineered to support the additional communication and I/O and must minimize task dispatch overheads. The hardware of large-scale HPC systems, with its high degree of parallelism and support for intensive communication, is well suited for MTC applications. However, HPC systems often lack a dynamic resource-provisioning feature, are not ideal for task communication via the file system, and have an I/O system that is not optimized for MTC-style applications. Hence, additional software support is likely to be required to gain full benefit from the HPC hardware.

💡 Research Summary

The paper provides a comprehensive overview of Many‑Task Computing (MTC) and examines how the forthcoming Blue Waters supercomputer—projected to be the largest NSF‑funded system when it enters production in 2012—can be configured to support this emerging workload class. MTC is defined as a collection of discrete tasks organized as a directed graph, where each node (task) has explicit input and output dependencies that form the edges. While this graph‑based structure resembles High‑Throughput Computing (HTC), the authors argue that MTC diverges from traditional HTC in several critical ways.

First, MTC applications are typically “time‑to‑solution” sensitive: they aim to produce results within seconds or minutes, even though the overall workload may consist of thousands to millions of very short tasks. This requirement forces the execution environment to minimize latency at every level, from task dispatch to data movement. Second, many MTC workflows rely heavily on file‑based communication; tasks read inputs and write outputs to a shared file system, creating a high volume of metadata operations and small‑file I/O. Third, because individual tasks are often lightweight (lasting only a few seconds), the overhead of the scheduler or resource manager can dominate overall performance. Consequently, an MTC‑friendly platform must provide (1) ultra‑low‑overhead task dispatch, (2) low‑latency inter‑task communication, and (3) an I/O subsystem optimized for massive concurrent small‑file accesses.

The authors then assess Blue Waters’ hardware characteristics. Its massive parallelism (hundreds of thousands of cores), high‑speed interconnect, and large memory capacity make it intrinsically capable of handling communication‑intensive workloads. However, the paper points out three mismatches between current HPC system designs and MTC needs. Traditional HPC resource managers employ static allocation: users request a fixed set of nodes for the entire job, which prevents the dynamic scaling that many MTC workloads would benefit from. The file system, typically a parallel Lustre or GPFS deployment, is tuned for large, sequential I/O streams rather than the bursty, metadata‑heavy pattern of MTC. Finally, the job launch mechanisms (e.g., batch schedulers) introduce non‑trivial latency when launching thousands of short tasks.

To bridge these gaps, the authors propose two complementary software strategies. The first is a “task‑scheduling layer” that sits atop existing batch systems but provides microsecond‑scale dispatch, in‑memory task queues, and dynamic resource reallocation. By keeping task descriptors in a fast key‑value store rather than on disk, the system reduces metadata traffic and enables rapid re‑scheduling when nodes become available or fail. The second strategy focuses on “data locality enhancement.” Instead of relying on the global file system for every data exchange, the approach pre‑stages inputs onto node‑local storage (SSD or RAM) and uses high‑performance message‑passing (MPI, RDMA) for intermediate results. When file I/O is unavoidable, the authors suggest adopting a streaming or object‑store model that aggregates many small files into larger containers, thereby alleviating metadata bottlenecks.

Both strategies are designed to be compatible with existing HPC middleware through well‑defined APIs, allowing incremental deployment without a complete system redesign. The paper emphasizes that successful MTC execution on Blue Waters will require (1) dynamic provisioning of compute resources, (2) a lightweight, latency‑optimized scheduler, and (3) an I/O stack that can efficiently handle the high‑frequency, small‑size data accesses characteristic of MTC.

In conclusion, the authors view MTC as a distinct workload class that sits between traditional HPC (large, tightly coupled simulations) and HTC (large numbers of loosely coupled jobs). Blue Waters possesses the raw hardware capability to excel at MTC, but realizing this potential demands targeted software enhancements and policy adjustments. By implementing dynamic resource management, low‑overhead task dispatch, and I/O optimizations, Blue Waters can become a premier platform not only for classic large‑scale simulations but also for the emerging generation of many‑task scientific workflows.