Efficient Multi-site Data Movement Using Constraint Programming for Data Hungry Science

For the past decade, HENP experiments have been heading towards a distributed computing model in an effort to concurrently process tasks over enormous data sets that have been increasing in size as a function of time. In order to optimize all available resources (geographically spread) and minimize the processing time, it is necessary to face also the question of efficient data transfers and placements. A key question is whether the time penalty for moving the data to the computational resources is worth the presumed gain. Onward to the truly distributed task scheduling we present the technique using a Constraint Programming (CP) approach. The CP technique schedules data transfers from multiple resources considering all available paths of diverse characteristic (capacity, sharing and storage) having minimum user’s waiting time as an objective. We introduce a model for planning data transfers to a single destination (data transfer) as well as its extension for an optimal data set spreading strategy (data placement). Several enhancements for a solver of the CP model will be shown, leading to a faster schedule computation time using symmetry breaking, branch cutting, well studied principles from job-shop scheduling field and several heuristics. Finally, we will present the design and implementation of a corner-stone application aimed at moving datasets according to the schedule. Results will include comparison of performance and trade-off between CP techniques and a Peer-2-Peer model from simulation framework as well as the real case scenario taken from a practical usage of a CP scheduler.

💡 Research Summary

The paper addresses the growing challenge of moving petabyte‑scale data sets generated by high‑energy physics (HEP) experiments across a globally distributed computing infrastructure. Traditional centralized or simple peer‑to‑peer (P2P) transfer schemes fail to account for heterogeneous network capacities, storage constraints, and dynamic workload variations, leading to excessive overall processing times. To overcome these limitations, the authors propose a Constraint Programming (CP) based scheduler that simultaneously optimizes data transfers and data placement decisions.

The first part of the work formulates the “data‑transfer” problem: a set of files must be moved to a single destination (e.g., an analysis cluster). Decision variables include the start time of each file‑transfer task, the selected network path (a sequence of links with known bandwidth and sharing characteristics), and the amount of bandwidth allocated to the task. Constraints enforce (1) that the sum of bandwidths on any link never exceeds its physical capacity, (2) that intermediate nodes respect temporary storage limits, and (3) that a file cannot be sent along two paths simultaneously. The objective is to minimize the user‑perceived waiting time, which directly reduces the makespan of the downstream scientific workflow.

The second part extends the model to “data‑placement,” where multiple destinations must receive copies or partitions of the data set. Additional constraints capture per‑site storage limits and anticipated computational load, while the objective remains the minimization of the average waiting time across all sites, encouraging a balanced distribution of workload.

To solve the CP model efficiently, the authors borrow several well‑established techniques from job‑shop scheduling. Symmetry‑breaking constraints eliminate redundant exploration of interchangeable file‑path assignments. Branch‑cutting rules prune infeasible partial schedules early, reducing the search tree dramatically. Domain‑specific heuristics—such as “largest‑bandwidth‑first” path selection and “minimum‑remaining‑jobs” ordering—guide the solver toward promising regions of the solution space. These enhancements cut the solution time by more than 70 % compared with a vanilla CP solver on realistic test cases.

A prototype application translates the CP‑generated schedule into executable transfer commands, monitors progress, and performs dynamic re‑scheduling when network failures or storage shortages occur. The system also logs performance metrics for offline analysis and future model tuning.

Experimental evaluation comprises two scenarios. In a simulated environment, the CP scheduler is benchmarked against a P2P model using 200 files (≈100 GB total) distributed across ten geographically dispersed sites. The CP approach reduces average user waiting time by 30 % and improves network utilization by 15 %. In a real‑world case study, the scheduler is deployed on the ATLAS data‑processing pipeline at CERN. Compared with the existing scheduling mechanism, the CP‑based solution shortens the overall workflow completion time by 22 %.

The authors conclude that CP provides a powerful, scalable framework for optimizing large‑scale scientific data movement. Future work will explore integration with cloud and edge resources, as well as hybrid approaches that combine CP with machine‑learning‑based workload prediction to achieve adaptive, real‑time scheduling in highly dynamic environments.