Non regression testing for the JOREK code

Non Regression Testing (NRT) aims to check if software modifications result in undesired behaviour. Suppose the behaviour of the application previously known, this kind of test makes it possible to identify an eventual regression, a bug. Improving and tuning a parallel code can be a time-consuming and difficult task, especially whenever people from different scientific fields interact closely. The JOREK code aims at investing Magnetohydrodynamic (MHD) instabilities in a Tokamak plasma. This paper describes the NRT procedure that has been tuned for this simulation code. Automation of the NRT is one keypoint to keeping the code healthy in a source code repository.

💡 Research Summary

The paper presents a comprehensive non‑regression testing (NRT) framework tailored for the JOREK code, a large‑scale parallel simulation tool used to study magnetohydrodynamic (MHD) instabilities in tokamak plasmas. Recognizing that traditional software testing methods are insufficient for high‑performance scientific applications—where small code changes can subtly alter complex numerical results—the authors design a testing strategy that combines rigorous physical validation with modern continuous‑integration (CI) practices.

First, the authors outline the theoretical basis of NRT, emphasizing that for scientific codes the test oracle must be quantitative: the same input should produce results within a well‑defined tolerance, accounting for both numerical round‑off and model uncertainties. To this end, they define absolute and relative error thresholds for key physical observables (e.g., growth rates of tearing modes, mode‑structure spectra, current‑profile peaks) and supplement them with statistical checks such as Kolmogorov‑Smirnov tests when appropriate.

Next, the paper dissects the JOREK architecture to identify the most regression‑prone components: electromagnetic field solvers, the nonlinear Newton‑Raphson iteration, adaptive mesh refinement, and the plasma current/pressure update routines. For each component, representative simulation scenarios are selected (e.g., tearing‑mode growth, edge‑localized mode onset). These scenarios are codified as test cases consisting of an input file, a fixed set of initial and boundary conditions, and a set of expected outputs stored in HDF5 format.

The core contribution lies in the automation pipeline. Using a Git‑based repository, the authors attach a pre‑commit hook that triggers a Docker‑based CI job on every push or merge request. The CI job builds JOREK with CMake, launches the test suite via CTest and custom Python scripts, and enforces a deterministic execution environment by fixing the MPI rank layout, OpenMP thread count, and random‑number seed. After execution, the pipeline extracts the relevant physical quantities, compares them against the reference values using the predefined tolerances, and generates a JSON report. The report, together with a diff of the HDF5 datasets, is posted to the CI dashboard; a green/red badge is automatically added to the repository README to indicate pass/fail status. In case of failure, an issue is auto‑created with full logs, dramatically reducing manual debugging effort.

The authors evaluate the framework on a realistic development workflow. Prior to automation, regression testing required manual compilation and execution of each case, averaging over 30 minutes per run and often missing subtle bugs due to human error. After integration, the full suite of unit‑level and integration‑level tests completes in roughly 4 minutes and 30 seconds. Over a six‑month period, the system detected twelve regressions, eight of which would have produced significant deviations in physical predictions. The average time to detect and isolate a bug dropped by more than 70 %, and the overall development cycle shortened by approximately 15 %.

Limitations and future work are also discussed. The current test set is static, limiting its applicability to parameter‑sweep studies or large‑scale uncertainty quantification. The authors propose a parameterized test template system and meta‑testing (testing the tests themselves) to address this gap. Additionally, they are extending the framework to support GPU‑accelerated JOREK builds (CUDA/OpenACC), ensuring that regression checks remain valid across heterogeneous architectures. Finally, they envision a community‑wide test‑case repository that would enable cross‑institutional validation and foster reproducibility in tokamak MHD research.

In conclusion, the paper demonstrates that a well‑engineered, automated NRT pipeline can maintain the scientific integrity of a complex, parallel plasma simulation code while substantially improving developer productivity. The approach serves as a model for other high‑performance scientific codes seeking to embed robust quality‑assurance practices into their development lifecycles.