Traceability and Provenance in Big Data Medical Systems

Providing an appropriate level of accessibility to and tracking of data or process elements in large volumes of medical data, is an essential requirement in the Big Data era. Researchers require systems that provide traceability of information through provenance data capture and management to support their clinical analyses. We present an approach that has been adopted in the neuGRID and N4U projects, which aimed to provide detailed traceability to support research analysis processes in the study of biomarkers for Alzheimers disease, but is generically applicable across medical systems. To facilitate the orchestration of complex, large-scale analyses in these projects we have adapted CRISTAL, a workflow and provenance tracking solution. The use of CRISTAL has provided a rich environment for neuroscientists to track and manage the evolution of data and workflow usage over time in neuGRID and N4U.

💡 Research Summary

This paper addresses the critical challenge of ensuring traceability and provenance in big data medical research systems, where reproducibility and verification of complex analyses are paramount. It presents a concrete implementation within the neuGRID and neuGRID for Users (N4U) projects, which focus on Alzheimer’s disease biomarker discovery through neuroimaging analysis.

The authors argue that as medical data volumes and analytical complexity grow, simply capturing the final result is insufficient. Researchers need a complete, queryable record of the “who, what, when, and how” of each analysis—the provenance. This includes the specific datasets used, the exact versions and parameters of algorithms applied, the sequence of processing steps, the computational resources involved, and the individuals responsible. This provenance data is essential for validating findings, reproducing studies, debugging errors, and facilitating collaboration.

To meet this need, the paper details the adoption and adaptation of CRISTAL, a provenance and workflow tracking system originally developed at CERN for managing the construction of the CMS particle detector. The core innovation of CRISTAL is its “description-driven” architecture. System elements (data, processes, agents) are modeled as “Items,” which have a lifecycle and a separation between their definition (a template) and their runtime instances. This allows for the dynamic, on-the-fly reconfiguration of workflows—a crucial feature for evolving scientific domains like neuroscience. Even while an analysis is running, researchers can modify pipeline parameters or structures. CRISTAL automatically versions all changes, preserving a complete historical trace while enabling new versions to operate concurrently with old ones.

Within the N4U Virtual Laboratory, CRISTAL serves as the provenance engine for an integrated platform called the “Analysis Base.” This base acts as a knowledge graph, storing and interlinking metadata about datasets, pipeline definitions, analysis execution instances, results, and users. The surrounding Virtual Laboratory provides services for data persistence, querying, workflow authoring, job dispatch to grid computing resources, and user interaction via a Science Gateway.

In operation, a researcher uses the Analysis Service to select a dataset and an analytical pipeline from the Analysis Base to create a new “Analysis” Item. The system then spawns child analysis instances for each data element, generating computational jobs for the grid. Throughout execution, CRISTAL meticulously logs provenance: workflow specifications, inputs/outputs for each step, inter-component dependencies, execution errors, resource usage, and timestamps. This creates an immutable, auditable trail for every piece of data and every process.

The paper concludes that the CRISTAL-based approach, as demonstrated in neuGRID/N4U, provides a generic, scalable, and reconfigurable solution for provenance management. It moves beyond simple workflow logging to create a collaborative research environment where past analyses are fully reproducible, reusable, and understandable, thereby strengthening the scientific rigor of big data medical research.