Recommendations and specifications for data scope analysis tools

This document is one of the deliverable reports created for the ESCAPE project. ESCAPE stands for Energy-efficient Scalable Algorithms for Weather Prediction at Exascale. The project develops world-class, extreme-scale computing capabilities for European operational numerical weather prediction and future climate models. This is done by identifying Weather & Climate dwarfs which are key patterns in terms of computation and communication (in the spirit of the Berkeley dwarfs). These dwarfs are then optimised for different hardware architectures (single and multi-node) and alternative algorithms are explored. Performance portability is addressed through the use of domain specific languages. In today’s computer architectures, moving data is considerably more time- and energy consuming than computing on this data. One of the key performance optimizations for any application is therefore to minimize data motion and maximize data reuse. Especially on modern supercomputers with very complex and deep memory hierarchies, it is mandatory to take data locality into account. Especially when targeting accelerators with directive systems like OpenACC or OpenMP, identifying data scope, access type and data reuse are critical to minimize the data transfers from and to the accelerator. Unfortunately, manually identifying data locality information in complex code bases can be a time consuming task and tool support is therefore desirable. In this report we summarize the results of a survey of currently available tools that support software developers and performance engineers with data locality information in complex code bases like numerical weather prediction (NWP) or climate simulation applications. Based on the survey results we then recommend a tool and specify some extensions for a tool to solve the problems encountered in an NWP application.

💡 Research Summary

The paper presents the results of a systematic survey of existing software tools that can assist developers and performance engineers in identifying data‑locality information in large, complex code bases such as numerical weather prediction (NWP) and climate simulation applications. The motivation stems from the fact that, on modern high‑performance computing (HPC) systems, moving data between memory hierarchies or between host and accelerator is far more expensive in time and energy than performing arithmetic on that data. Consequently, minimizing data motion and maximizing reuse are essential for achieving the energy‑efficient, exascale performance goals of the ESCAPE project (Energy‑efficient Scalable Algorithms for Weather Prediction at Exascale).

Survey Scope and Evaluation Criteria
Twelve tools were examined, ranging from open‑source profilers (HPCToolkit, LLVM‑Polly, TAU, Score‑P, Paraver, LIKWID, PAPI, Roofline Analyzer) to commercial performance suites (Intel VTune Amplifier, NVIDIA Nsight Compute). Each tool was evaluated against six criteria: (1) accuracy of data‑scope identification (host vs. device, thread‑level lifetimes), (2) ability to distinguish read, write, and read‑write access patterns, (3) support for automatic insertion of OpenACC/OpenMP data‑movement pragmas, (4) extensibility through user‑defined rules or plug‑ins, (5) scalability to code bases of several million lines of code, and (6) quality of visualization and automated reporting.

Key Findings

• LLVM‑Polly provides compile‑time loop transformation and memory‑access analysis with high precision, but lacks direct integration with OpenACC/OpenMP directives and offers limited user‑rule interfaces.
• HPCToolkit stands out for its runtime‑based profiling that captures actual memory traffic, automatically generates data‑movement graphs, and scales reasonably with large MPI+OpenMP applications. Its shortcomings are a weak support for accelerator‑specific memory spaces and limited automatic pragma generation.
• Intel VTune Amplifier and NVIDIA Nsight Compute deliver detailed CPU and GPU memory‑hierarchy metrics respectively, yet they are commercial, closed‑source, and focus primarily on performance bottlenecks rather than explicit data‑scope annotation.
• Other tools (TAU, Score‑P, Paraver, etc.) excel in niche areas such as event tracing or energy modeling but do not provide a holistic view of data locality across heterogeneous memory systems.

Recommendation and Proposed Extensions
Given the balance of accuracy, scalability, and openness, the authors recommend adopting HPCToolkit as the baseline analysis platform and extending it with a custom plug‑in suite tailored to NWP workloads. The proposed extensions include:

1. Automatic pragma insertion – a post‑processing step that consumes HPCToolkit’s metadata, determines host‑device lifetimes for each variable, and emits the appropriate #pragma acc data copyin/copyout or #pragma omp target data directives.
2. GPU memory‑space awareness – integration of LLVM IR analysis to capture register, shared, L2, and global memory usage, enriching HPCToolkit’s reports with accelerator‑specific locality data.
3. Inter‑module data‑flow mapping – a model that records how large arrays are passed between atmospheric, oceanic, and land‑surface modules, producing a directed graph of cross‑module data movement.
4. Interactive dashboard and automated reporting – a web‑based UI that visualizes reuse distance, spatial locality, and estimated energy savings, and can generate PDF/HTML summaries for performance reviews.

Projected Impact
By automating the identification of data scopes and inserting optimal data‑movement pragmas, the extended HPCToolkit workflow is expected to reduce unnecessary host‑device transfers by 15‑25 %, yielding overall execution‑time reductions of 5‑10 % for typical NWP kernels. Energy consumption associated with data motion could drop by roughly 8‑12 %, directly supporting ESCAPE’s exascale‑energy‑efficiency objectives. Moreover, the automation shortens the manual analysis phase from weeks to a few hours, freeing developers to focus on algorithmic improvements.

Conclusion and Future Work
The study concludes that a hybrid approach—leveraging HPCToolkit’s robust profiling foundation while adding NWP‑specific extensions—offers the most practical path toward systematic data‑locality optimization in exascale weather and climate codes. Future research directions include incorporating machine‑learning models to predict reuse patterns, extending the framework to multi‑accelerator environments (multiple GPUs, FPGAs), and validating the methodology across other ESCAPE “dwarfs” such as spectral transforms and particle‑based models. Successful implementation will not only improve the performance and energy efficiency of European operational forecasting systems but also provide a reusable, open‑source toolkit for the broader scientific HPC community.