Scientific Workflow Systems for 21st Century e-Science, New Bottle or New Wine?

With the advances in e-Sciences and the growing complexity of scientific analyses, more and more scientists and researchers are relying on workflow systems for process coordination, derivation automation, provenance tracking, and bookkeeping. While workflow systems have been in use for decades, it is unclear whether scientific workflows can or even should build on existing workflow technologies, or they require fundamentally new approaches. In this paper, we analyze the status and challenges of scientific workflows, investigate both existing technologies and emerging languages, platforms and systems, and identify the key challenges that must be addressed by workflow systems for e-science in the 21st century.

💡 Research Summary

The paper surveys the state of scientific workflow systems (SWFS) in the context of modern e‑Science and asks whether they should be built on legacy workflow technologies or require fundamentally new approaches. It begins by defining the four core functions of a scientific workflow: (1) describing complex scientific procedures, (2) automating data derivation, (3) leveraging high‑performance computing (HPC) to improve throughput, and (4) managing provenance. The authors note that while workflow concepts have existed since the 1980s, the explosion of data size, the rise of multicore processors, and the convergence of supercomputing and grid infrastructures have dramatically altered the requirements for SWFS.

Section 2 discusses multicore architectures. With chip manufacturers moving from frequency scaling to core scaling, systems with dozens to hundreds of cores are now commonplace, and projections suggest thousands of cores per node within a decade. This shift forces software designers to expose parallelism explicitly or implicitly; traditional sequential workflow engines cannot efficiently exploit such massive concurrency.

Section 3 addresses the “data deluge” problem. Scientific domains such as astronomy (SDSS), high‑energy physics (CERN’s CMS), and bioinformatics (GenBank, EMBL) generate terabytes to petabytes of data annually. Data movement often dominates execution time, making data locality a critical design factor. The authors argue that workflow runtimes must incorporate data‑aware scheduling, intelligent caching, and replication strategies so that tasks are placed near the data they consume, thereby reducing network bottlenecks.

Section 4 compares supercomputing and grid computing. Historically, supercomputers were the sole platform for compute‑intensive tasks, requiring specialized programming models (MPI, OpenMP). Grids, built from commodity clusters, offered more flexible, loosely‑coupled execution across administrative domains. However, modern supercomputers now employ multicore nodes and run standard Linux environments, blurring the line between the two. Consequently, a robust SWFS should be able to schedule work transparently across both grids and supercomputers, adapting to the characteristics of each resource pool.

Section 5 reviews existing and emerging workflow technologies. Traditional DAG‑based systems such as DAGMan and Pegasus dominate grid environments. Domain‑specific tools like Taverna (bioinformatics) and Kepler (visual modeling) provide rich graphical interfaces but are limited in scalability. VisTrails adds workflow versioning for exploratory science. Emerging paradigms include MapReduce (simple map/reduce model for massive data processing), Fortress (a high‑level language designed for HPC with implicit parallelism), Microsoft Windows Workflow Foundation (application‑level orchestration), Star‑P (language extensions for MATLAB/Python/R), and Swift (a scripting language plus a runtime that integrates CoG Karajan and the lightweight Falkon execution service). Each system addresses a subset of the required capabilities, but none offers a complete solution.

In the final section, the authors issue a call for next‑generation scientific workflow systems. They stress the need for languages that expose parallelism either explicitly or implicitly, allowing compilers to infer data dependencies and generate massive numbers of concurrent tasks. Data‑aware scheduling, dynamic resource provisioning, and integrated provenance tracking must be baked into the runtime. Moreover, the system should provide a modular “workflow bus” that can interoperate with existing engines, thereby leveraging mature components while extending functionality. The paper concludes that scientific workflows are evolving from simple task orchestration to full‑scale distributed applications that must simultaneously manage data, computation, and reproducibility, and that only a new generation of workflow platforms can meet these 21st‑century e‑Science challenges.