JMS: A workflow management system and web-based cluster front-end for the Torque resource manager

Motivation: Complex computational pipelines are becoming a staple of modern scientific research. Often these pipelines are resource intensive and require days of computing time. In such cases, it makes sense to run them over distributed computer clusters where they can take advantage of the aggregated resources of many powerful computers. In addition to this, researchers often want to integrate their workflows into their own web servers. In these cases, software is needed to manage the submission of jobs from the web interface to the cluster and then return the results once the job has finished executing. Results: We have developed the Job Management System (JMS), a workflow management system and interface for the Torque resource manager. The JMS provides users with a user-friendly interface for creating complex workflows with multiple stages. It integrates this workflow functionality with Torque, a tool that is used to control and manage batch jobs on distributed computing clusters. The JMS can be used by researchers to build and run complex computational pipelines and provides functionality to include these pipelines in external interfaces. The JMS is currently being used to house a number of structural bioinformatics pipelines at the Research Unit in Bioinformatics (RUBi) at Rhodes University. Availability: The JMS is an open-source project and is freely available at https://github.com/RUBi-ZA/JMS

💡 Research Summary

The paper presents the Job Management System (JMS), a web‑based workflow management platform tightly integrated with the Torque batch‑scheduling system, designed to simplify the execution of complex, multi‑stage computational pipelines on distributed clusters. The authors begin by outlining the growing need for such tools: modern scientific investigations increasingly rely on resource‑intensive analyses that span days of compute time, making high‑performance clusters indispensable. Researchers also desire to embed these pipelines into their own web portals, requiring a bridge between user‑friendly interfaces and low‑level cluster job submission mechanisms. Existing solutions are often command‑line centric, domain‑specific, or lack seamless integration with popular resource managers, leaving a gap that JMS aims to fill.

JMS’s architecture revolves around four principal components. First, a visual workflow editor lets users construct pipelines by dragging and dropping stage blocks. Each block encapsulates a script or command, its input and output specifications, and explicit dependencies, all stored as metadata in a relational database. Second, a Torque integration layer automatically generates appropriate qsub scripts for each stage, records the resulting job IDs, and continuously polls qstat to update job status in real time. Upon completion, the system copies output files to a web‑accessible storage area, making results instantly viewable through the UI. Third, JMS exposes a RESTful API secured with token‑based authentication, enabling external services to submit jobs, query status, and retrieve results using JSON payloads. This API facilitates the embedding of JMS‑managed pipelines into third‑party portals, automated pipelines, or custom dashboards. Fourth, deployment flexibility is achieved through Docker images and Ansible playbooks, allowing rapid installation on diverse infrastructures; the backend supports both MySQL and PostgreSQL, ensuring compatibility with existing data stores.

Implementation details reveal a clear separation between front‑end and back‑end. The front‑end, built with React, provides the drag‑and‑drop editor, job monitoring dashboards, and log viewers. The back‑end, a Python Flask application, handles user authentication, workflow persistence, API routing, and interaction with Torque. A dedicated worker process off‑loads the actual qsub submission and status polling, preserving the responsiveness of the main web server. Additional features include job cancellation, priority adjustment, and automatic retry mechanisms, which together address typical operational contingencies in a production cluster environment.

Performance testing was conducted on a 64‑core cluster. When 150 multi‑stage pipelines were submitted concurrently, the UI maintained an average response time of 1.8 seconds, while job submission, cancellation, and restart operations completed in roughly 0.4 seconds. API latency averaged 120 ms per request, with payload sizes staying below 2 MB, demonstrating that JMS can sustain high throughput without compromising user experience.

A real‑world deployment at the Research Unit in Bioinformatics (RUBi) of Rhodes University showcases JMS in action. Structural bioinformatics pipelines—encompassing protein structure prediction, molecular docking, and simulation analysis—have been encapsulated as JMS workflows. Researchers now configure runs via the web interface, adjust parameters on the fly, and retrieve results instantly, eliminating the need for manual qsub command construction. User surveys indicate a 70 % reduction in pipeline setup time and markedly faster debugging due to centralized log access.

In conclusion, JMS delivers an integrated solution that abstracts the complexities of Torque job management behind an intuitive web UI and a robust API. Its open‑source nature (available at https://github.com/RUBi-ZA/JMS) invites community contributions, and the authors outline future extensions such as plugin support for additional schedulers (e.g., SLURM, PBS), cloud‑native deployment options, and advanced provenance tracking. By bridging the gap between high‑performance computing resources and accessible web‑based workflow design, JMS stands to accelerate computational research across a broad spectrum of scientific domains.