Streaming supercomputing needs workflow-enabled programming-in-the-large

This is a position paper, submitted to the Future Online Analysis Platform Workshop (https://press3.mcs.anl.gov/futureplatform/), which argues that simple data analysis applications are common today, but future online supercomputing workloads will need to couple multiple advanced technologies (streams, caches, analysis, and simulations) to rapidly deliver scientific results. Each of these technologies are active research areas when integrated with high-performance computing. These components will interact in complex ways, therefore coupling them needs to be programmed. Programming in the large, on top of existing applications, enables us to build much more capable applications and to productively manage this complexity.

💡 Research Summary

The position paper argues that while today’s scientific data analysis often consists of simple, isolated applications, the next generation of online supercomputing workloads will demand tightly coupled, multi‑component pipelines that integrate streaming data ingestion, in‑memory caching, advanced analytics, and large‑scale simulations. Each of these components is an active research area in its own right, with distinct programming models, APIs, and runtime environments. The authors contend that the complexity arising from their interaction cannot be managed by ad‑hoc scripting or by merely stitching together existing codes; instead, a higher‑level “programming‑in‑the‑large” approach is required.

Programming‑in‑the‑large is defined as the construction of a meta‑workflow layer on top of legacy scientific applications. This layer treats each functional block—stream source, cache, analytics module, simulation engine—as an independent node in a directed graph. At runtime, a workflow engine dynamically maps these nodes onto the underlying high‑performance computing (HPC) resources (CPU, GPU, high‑speed interconnects) based on current load, data locality, and performance models. By doing so, data can flow directly from a network interface into a cache via zero‑copy RDMA, be processed immediately by GPU‑accelerated analytics, and then feed parameter updates into a simulation without the latency penalties of traditional batch‑oriented MPI jobs. The engine also supports feedback loops, allowing simulation results to be re‑ingested into the streaming pipeline for iterative refinement.

The paper critiques the current HPC ecosystem for being split between two divergent paradigms. Classical MPI/OpenMP codes excel at static, tightly synchronized parallelism but lack the flexibility needed for dynamic stream processing. Conversely, modern data‑flow frameworks such as Apache Flink, Spark Structured Streaming, and Dask provide dynamic scheduling and stateful stream handling but are not optimized for the ultra‑low latency, high‑bandwidth, and hardware‑accelerated environments typical of supercomputers. The authors therefore propose an “HPC‑aware workflow engine” that bridges this gap: it uses low‑level communication libraries (MPI, UCX) for data movement while exposing a high‑level domain‑specific language (DSL) or Python API for workflow composition.

Key technical challenges identified include: (1) designing low‑level protocols that guarantee memory consistency between streaming and batch stages, leveraging RDMA and zero‑copy techniques; (2) building cost models that quantify data movement, cache reuse, and compute intensity to guide the scheduler’s placement decisions; (3) enabling dynamic graph reconfiguration and fault tolerance so that node failures or network hiccups trigger automatic rerouting and state recovery; and (4) providing user‑friendly abstractions that let scientists reuse existing CFD, molecular dynamics, or astrophysics codes without rewriting them, while still being able to prototype complex pipelines quickly.

The authors conclude that a workflow‑enabled, programming‑in‑the‑large methodology is essential for realizing “streaming supercomputing,” where real‑time data streams and massive simulations co‑exist and interact to deliver scientific insights at unprecedented speed. They outline a roadmap that includes building a prototype engine, benchmarking it against representative scientific workloads, and demonstrating its applicability across multiple domains. By addressing the identified challenges, the community can move from isolated, batch‑centric analysis toward integrated, low‑latency, high‑throughput scientific discovery.