Data Preservation in High Energy Physics - why, how and when?

Long-term preservation of data and software of large experiments and detectors in high energy physics is of utmost importance to secure the heritage of (mostly unique) data and to allow advanced physics (re-)analyses at later times. Summarising the work of an international study group, motivation, use cases and technical details are given for an organised effort to secure and enable future use of past, present and future experimental data. As a practical use case and motivation, the revival of JADE data and the corresponding latest results on measuring $\alpha_s$ in NNLO QCD are reviewed.

💡 Research Summary

The paper addresses the critical need to preserve the data and software of large‑scale high‑energy physics (HEP) experiments for the long term, arguing that such preservation is essential to protect a unique scientific heritage and to enable future physics analyses that can benefit from advances in theory and computing. Drawing on the work of the international DPHEP (Data Preservation in High Energy Physics) study group, the authors first outline the motivations for preservation: safeguarding irreplaceable collision data, allowing re‑interpretation of results with newer theoretical models, providing educational and outreach material, and fostering cross‑experiment collaboration. They then present concrete use cases, such as cross‑experiment comparisons, application of modern machine‑learning techniques to legacy data, systematic uncertainty re‑evaluation with updated detector calibrations, and the creation of teaching data sets for university courses.

The paper adopts the DPHEP four‑level preservation model. Level 1 covers documentation only; Level 2 adds analysis‑level data formats (e.g., ROOT files) and basic analysis scripts; Level 3 includes the full analysis workflow, typically encapsulated in containers or workflow description languages; Level 4 represents the most ambitious scenario, preserving raw detector data together with the complete software stack, compilers, and operating system. The authors discuss the increasing cost, manpower, and technical complexity associated with each higher level and recommend that experiments select a level that matches their expected future usage.

Technical solutions are described in detail. Rich metadata schemas are advocated to capture detector conditions, calibration constants, and provenance information, enabling efficient search and reuse. Sustainable file formats such as ROOT and HDF5 are preferred, and conversion tools are provided to mitigate future format obsolescence. Virtualisation and containerisation (Docker, Singularity) are highlighted as key to freezing software environments, ensuring that code written decades ago can still be executed on modern hardware. Long‑term storage strategies combine tape libraries with cloud‑based object storage to balance cost, durability, and accessibility. Governance structures are proposed, including dedicated preservation teams within each collaboration, early‑stage Data Management Plans, diversified funding streams (national agencies, international consortia, industry partnerships), and formal recognition of preservation work as a scientific output.

A central case study is the revival of the JADE experiment data from the PETRA e⁺e⁻ collider. The original data were stored in proprietary formats and processed with FORTRAN programs tied to obsolete hardware. By migrating the raw files to modern ROOT trees, encapsulating the original analysis software in Docker images, and linking the data to contemporary NNLO QCD calculations, the authors were able to re‑measure the strong coupling constant αₛ(M_Z) with significantly reduced uncertainties. This achievement demonstrates that legacy data, when properly preserved, can contribute to precision tests of the Standard Model that were impossible at the time of collection. The JADE experience also yielded practical lessons: the necessity of comprehensive metadata, the importance of version‑controlled software archives, and the need for periodic integrity checks of storage media.

In conclusion, the authors argue that data preservation should be viewed not merely as archiving but as the creation of a reusable scientific asset. They propose a roadmap that combines automated metadata capture, AI‑assisted data quality monitoring, and an international data portal to provide unified access to preserved HEP datasets. By implementing these strategies, the HEP community can ensure that past, present, and future experiments continue to generate scientific value long after the detectors have been shut down.