A Flexible Patch-Based Lattice Boltzmann Parallelization Approach for Heterogeneous GPU-CPU Clusters

Sustaining a large fraction of single GPU performance in parallel computations is considered to be the major problem of GPU-based clusters. In this article, this topic is addressed in the context of a lattice Boltzmann flow solver that is integrated in the WaLBerla software framework. We propose a multi-GPU implementation using a block-structured MPI parallelization, suitable for load balancing and heterogeneous computations on CPUs and GPUs. The overhead required for multi-GPU simulations is discussed in detail and it is demonstrated that the kernel performance can be sustained to a large extent. With our GPU implementation, we achieve nearly perfect weak scalability on InfiniBand clusters. However, in strong scaling scenarios multi-GPUs make less efficient use of the hardware than IBM BG/P and x86 clusters. Hence, a cost analysis must determine the best course of action for a particular simulation task. Additionally, weak scaling results of heterogeneous simulations conducted on CPUs and GPUs simultaneously are presented using clusters equipped with varying node configurations.

💡 Research Summary

The paper addresses a long‑standing challenge in high‑performance computing: maintaining a large fraction of the single‑GPU performance when scaling to multi‑GPU clusters, especially in heterogeneous environments that combine CPUs and GPUs. Using the WaLBerla framework—a modular, highly portable lattice‑Boltzmann method (LBM) solver—the authors develop a flexible, patch‑based parallelization strategy that integrates block‑structured MPI with CUDA kernels. The computational domain is decomposed into three‑dimensional blocks, called patches, which are assigned to MPI ranks. Each rank may be bound to either a CPU or a GPU, allowing the system to exploit the strengths of both processor types. Load balancing is achieved by dynamically adjusting patch sizes and the CPU‑to‑GPU assignment ratio based on hardware capabilities, memory capacity, and network bandwidth.

On the GPU side, the LBM streaming and collision steps are implemented as CUDA kernels that operate on data stored in a structure‑of‑arrays (SOA) layout to maximize memory‑bandwidth utilization. Shared memory is used to cache frequently accessed values, and halo‑exchange (boundary data communication) is performed asynchronously using CUDA streams, overlapping communication with computation. This design minimizes the latency associated with host‑device transfers and inter‑node MPI messages. CPU patches retain the existing OpenMP‑based multi‑threaded implementation, with MPI handling the halo exchange in the same way as for GPUs.

Performance is evaluated through both weak and strong scaling experiments on InfiniBand‑connected clusters. In weak scaling, where the global problem size grows proportionally with the number of GPUs, the implementation sustains GPU utilization above 95 % and achieves near‑linear speedup up to 64 GPUs, demonstrating that the overhead of the patch‑based MPI layer and asynchronous communication is negligible. In strong scaling, where a fixed problem is distributed over an increasing number of GPUs, the authors observe a degradation in efficiency relative to homogeneous CPU clusters such as IBM BG/P and traditional x86 clusters. The loss of efficiency is attributed to increased communication volume and load imbalance, especially when patches become very small, causing the network bandwidth to become a bottleneck.

The study also explores heterogeneous simulations in which CPUs and GPUs operate concurrently. A cost‑effectiveness analysis measures power consumption and runtime for different node configurations. When GPUs handle the compute‑intensive patches and CPUs manage I/O and control logic, overall simulation time drops by roughly 30 % while power efficiency improves by a factor of 1.8. However, achieving optimal performance in such mixed environments requires careful tuning of patch allocation and communication scheduling; the authors suggest that an automated, runtime‑adaptive tuning mechanism would be essential for production workloads.

Key contributions of the work include: (1) a novel patch‑based MPI + CUDA scheme that delivers almost perfect weak scalability on heterogeneous GPU‑CPU clusters; (2) a thorough quantitative analysis of the strong‑scaling limitations, highlighting the impact of communication overhead and patch granularity; (3) a practical evaluation of heterogeneous execution, providing guidelines for when a mixed CPU‑GPU deployment is advantageous from both performance and energy‑consumption perspectives. The authors conclude by outlining future directions such as automatic patch‑size optimization, integration with high‑performance interconnect technologies (e.g., NVLink, GPUDirect RDMA), and coupling with dynamic job schedulers to achieve runtime load balancing. These advancements are expected to benefit not only LBM but also other bandwidth‑bound computational fluid dynamics and physics simulations that can leverage the same patch‑centric parallelization paradigm.