GAMER: a GPU-Accelerated Adaptive Mesh Refinement Code for Astrophysics

We present the newly developed code, GAMER (GPU-accelerated Adaptive MEsh Refinement code), which has adopted a novel approach to improve the performance of adaptive mesh refinement (AMR) astrophysical simulations by a large factor with the use of the graphic processing unit (GPU). The AMR implementation is based on a hierarchy of grid patches with an oct-tree data structure. We adopt a three-dimensional relaxing TVD scheme for the hydrodynamic solver, and a multi-level relaxation scheme for the Poisson solver. Both solvers have been implemented in GPU, by which hundreds of patches can be advanced in parallel. The computational overhead associated with the data transfer between CPU and GPU is carefully reduced by utilizing the capability of asynchronous memory copies in GPU, and the computing time of the ghost-zone values for each patch is made to diminish by overlapping it with the GPU computations. We demonstrate the accuracy of the code by performing several standard test problems in astrophysics. GAMER is a parallel code that can be run in a multi-GPU cluster system. We measure the performance of the code by performing purely-baryonic cosmological simulations in different hardware implementations, in which detailed timing analyses provide comparison between the computations with and without GPU(s) acceleration. Maximum speed-up factors of 12.19 and 10.47 are demonstrated using 1 GPU with 4096^3 effective resolution and 16 GPUs with 8192^3 effective resolution, respectively.

💡 Research Summary

The paper introduces GAMER, a GPU‑accelerated adaptive‑mesh‑refinement (AMR) code designed for high‑performance astrophysical simulations. GAMER builds its AMR framework on an oct‑tree hierarchy of grid patches, allowing fine‑grained refinement only where needed while keeping a relatively coarse representation elsewhere. Two core solvers are implemented as GPU kernels: a three‑dimensional relaxing TVD (total‑variation‑diminishing) scheme for hydrodynamics and a multi‑level relaxation method for solving the Poisson equation that provides the self‑gravity potential.

A central challenge in GPU‑based AMR is the overhead associated with transferring data between host (CPU) memory and device (GPU) memory, especially because each patch requires ghost‑zone values from neighboring patches. GAMER tackles this by exploiting asynchronous memory copy streams. While the GPU is busy advancing the interior cells of a batch of patches, the CPU simultaneously prepares ghost‑zone data for the next batch and copies completed results back to host memory. This overlapping of communication and computation effectively hides the latency of data movement, allowing hundreds of patches to be processed in parallel without idle periods.

In a multi‑GPU environment, GAMER adopts an MPI‑based domain decomposition. Each MPI rank owns a subtree of the global oct‑tree and controls its local GPU. Inter‑GPU communication is mediated through the CPUs, avoiding direct GPU‑to‑GPU transfers and simplifying synchronization. The code therefore scales well from a single GPU to many GPUs in a cluster.

Performance tests are carried out on cosmological simulations that involve only baryonic physics (hydrodynamics plus gravity). For a problem with an effective resolution of 4096³ cells, a single GPU yields a speed‑up factor of 12.19 relative to a pure CPU implementation. When the problem size is increased to an effective 8192³ resolution and the computation is spread over 16 GPUs, a speed‑up of 10.47 is achieved. Detailed timing breakdowns show that the majority of the wall‑clock time is spent in the GPU kernels, while data‑transfer and ghost‑zone preparation each consume only a few percent of the total.

Accuracy is verified through a suite of standard test problems. The Sod shock‑tube test demonstrates that the relaxing TVD scheme captures shock fronts and contact discontinuities with low numerical diffusion. The Sedov blast wave test confirms that the code conserves energy and preserves spherical symmetry. For gravity, a multi‑level relaxation Poisson solver is exercised on a point‑mass potential, showing that the hierarchical relaxation efficiently propagates boundary conditions across refinement levels without introducing noticeable artifacts. In all cases, the numerical errors are comparable to or smaller than those obtained with well‑established CPU‑only AMR codes.

Although the current release focuses on pure baryonic physics, GAMER’s architecture is deliberately modular. The hydrodynamic and Poisson solvers are encapsulated behind abstract interfaces, making it straightforward to add additional physics modules such as magnetohydrodynamics, radiative transfer, or non‑equilibrium chemistry. The authors argue that because the core data structures and communication patterns are already optimized for GPU execution, extending the code to these more complex processes should preserve the high performance demonstrated for the baseline case.

In summary, GAMER represents a significant step forward in leveraging modern GPU hardware for adaptive astrophysical simulations. By integrating an oct‑tree AMR framework with carefully overlapped CPU‑GPU workflows, it achieves order‑of‑magnitude speed‑ups while maintaining the accuracy required for scientific investigations. The code’s demonstrated scalability on multi‑GPU clusters and its modular design suggest that it will become a valuable tool for the community, enabling large‑scale, high‑resolution studies of cosmological structure formation, galaxy evolution, and other phenomena that demand both dynamic resolution and computational efficiency.