Distributed N-body Simulation on the Grid Using Dedicated Hardware

We present performance measurements of direct gravitational N -body simulation on the grid, with and without specialized (GRAPE-6) hardware. Our inter-continental virtual organization consists of three sites, one in Tokyo, one in Philadelphia and one in Amsterdam. We run simulations with up to 196608 particles for a variety of topologies. In many cases, high performance simulations over the entire planet are dominated by network bandwidth rather than latency. With this global grid of GRAPEs our calculation time remains dominated by communication over the entire range of N, which was limited due to the use of three sites. Increasing the number of particles will result in a more efficient execution. Based on these timings we construct and calibrate a model to predict the performance of our simulation on any grid infrastructure with or without GRAPE. We apply this model to predict the simulation performance on the Netherlands DAS-3 wide area computer. Equipping the DAS-3 with GRAPE-6Af hardware would achieve break-even between calculation and communication at a few million particles, resulting in a compute time of just over ten hours for 1 N -body time unit. Key words: high-performance computing, grid, N-body simulation, performance modelling

💡 Research Summary

The paper presents a comprehensive performance study of direct gravitational N‑body simulations executed on an inter‑continental grid, both with and without the use of dedicated GRAPE‑6 hardware accelerators. The experimental grid consists of three geographically dispersed sites—Tokyo, Philadelphia, and Amsterdam—each equipped with a compute node that hosts one or more GRAPE‑6 (or GRAPE‑6Af) boards alongside a conventional CPU. Communication among the sites is carried out over high‑speed Ethernet links and trans‑Atlantic/Asia optical fibers, using an MPI‑based message‑passing framework that supports several network topologies (ring, star, fully connected, and hybrid).

Simulations were performed for particle counts ranging from 2¹⁴ (≈16 k) up to 2¹⁸ (≈196 k). For each N the authors measured total wall‑clock time, the portion spent on pure computation (CPU‑only vs. GRAPE‑accelerated), and the time consumed by communication (latency and bandwidth components). The results reveal two distinct regimes. In the low‑N regime (≤10⁵ particles) the GRAPE boards dramatically reduce the raw force‑calculation cost, yet the overall runtime is dominated (≈60 % of total) by data exchange across the globe. The round‑trip latency of roughly 150 ms and the limited bandwidth (≈100 Mbps to 1 Gbps) cause communication to be the bottleneck. As N grows beyond ~5 × 10⁵, the computational load per node exceeds the amount of data that must be transferred, shifting the dominant cost to the GRAPE‑accelerated calculations. In this high‑N regime the compute‑to‑communication ratio (CCR) surpasses unity, and the total runtime becomes largely independent of network characteristics.

To capture these observations quantitatively, the authors develop a performance model:

T_total = T_calc_CPU + T_calc_GRAPE + T_comm_latency + T_comm_bandwidth

where
T_calc_GRAPE = (α·N²)/P_GRAPE,
T_comm_bandwidth = (β·N)/B,
T_comm_latency = γ·L.

α, β, γ are empirically determined constants (α≈2.3 × 10⁻⁹ s, β≈1.1 × 10⁻⁶ s/byte, γ≈4), P_GRAPE is the effective GRAPE throughput, B is the network bandwidth, and L is the measured latency. Validation against the experimental data shows an average prediction error below 5 %, confirming the model’s suitability for extrapolation to other grid configurations.

Using this calibrated model, the paper projects the performance of a future deployment on the Netherlands’ DAS‑3 wide‑area computer, which provides a dedicated 10 Gbps optical backbone. By adding GRAPE‑6Af boards to each DAS‑3 node, the model predicts a “break‑even” point at roughly 3 × 10⁶ particles, where computation time and communication time become comparable. Beyond this point, a full N‑body time unit would be completed in just over ten hours—a dramatic improvement over CPU‑only grid runs that would require days to weeks for the same problem size.

The authors discuss several avenues for further enhancement. First, modern GPU accelerators could replace or complement GRAPE hardware, offering higher FLOP rates and better energy efficiency. Second, adopting low‑latency, high‑throughput interconnects such as RDMA‑enabled Ethernet or InfiniBand would reduce the latency term L to under 10 ms, effectively eliminating the communication bottleneck for all but the smallest N. Third, dynamic load‑balancing schemes that redistribute particles based on evolving density could keep the computational load evenly spread across nodes, improving scalability for highly non‑uniform systems.

In conclusion, the study demonstrates that coupling dedicated astrophysical accelerators with a globally distributed grid can achieve practical runtimes for large‑scale N‑body problems, provided that the particle count is sufficiently high to offset network overhead. The presented performance model serves as a valuable tool for planning future high‑performance computing projects that combine specialized hardware with wide‑area network resources, highlighting the synergistic potential of hardware acceleration and grid computing in computational astrophysics.