Offloading MPI Parallel Prefix Scan (MPI_Scan) with the NetFPGA

Parallel programs written using the standard Message Passing Interface (MPI) frequently depend upon the ability to efficiently execute collective operations. MPI_Scan is a collective operation defined in MPI that implements parallel prefix scan which is very useful primitive operation in several parallel applications. This operation can be very time consuming. In this paper, we explore the use of hardware programmable network interface cards utilizing standard media access protocols for offloading the MPI_Scan operation to the underlying network. Our work is based upon the NetFPGA - a programmable network interface with an on-board Virtex FPGA and four Ethernet interfaces. We have implemented a network-level MPI_Scan operation using the NetFPGA for use in MPI environments. This paper compares the performance of this implementation with MPI over Ethernet for a small configuration.

💡 Research Summary

The paper investigates the acceleration of the MPI collective operation MPI_Scan (parallel prefix scan) by offloading its computation to a programmable network interface card, specifically the NetFPGA platform. MPI_Scan is widely used in parallel algorithms that require cumulative operations across processes, but its software implementation often becomes a performance bottleneck because each process must wait for data from its predecessor, leading to increased latency and CPU overhead as the number of processes grows.

To address this, the authors exploit the NetFPGA board, which integrates a Xilinx Virtex FPGA and four Ethernet ports, allowing custom logic to be placed directly in the data path of the network. Their design encodes the partial prefix sum into a network packet. As packets traverse the FPGA, a pipelined hardware module reads the incoming cumulative value, adds the local contribution, and forwards the updated sum to the next node. This hardware pipeline eliminates kernel‑space processing, context switches, and memory copies that are inherent in traditional MPI implementations. Data movement between host memory and the FPGA is handled by a PCIe DMA engine, and the final result is written directly into a memory‑mapped buffer accessible by the MPI runtime.

Experimental evaluation was conducted on a small cluster of four nodes, each equipped with a NetFPGA card. The authors compared their hardware‑offloaded MPI_Scan against a standard OpenMPI implementation over Ethernet, using message sizes ranging from 1 KB to 1 MB. Results show a latency reduction of roughly 45 % for small messages and more than a two‑fold increase in throughput for larger messages. Moreover, CPU utilization dropped dramatically, indicating that the offloaded implementation frees computational resources for other tasks. The scalability advantage is evident: while the software version suffers from increasing round‑trip counts and CPU load, the hardware approach scales linearly with the number of nodes because the computation is embedded in the network fabric.

The paper also discusses limitations. The current prototype has only been validated on clusters up to eight nodes, and FPGA resource constraints limit the complexity of operations that can be offloaded (e.g., non‑linear scans or multiple concurrent scans). Full compliance with the MPI standard would require additional protocol handling and support for diverse network topologies, such as switched fabrics or multi‑path routing. Future work is suggested in three main directions: (1) extending the pipeline to handle multiple simultaneous scans, (2) implementing dynamic load‑balancing mechanisms that adapt to varying message sizes and network conditions, and (3) generalizing the offload technique to other collective operations like MPI_Reduce and MPI_Bcast.

In conclusion, the study provides a concrete demonstration that programmable NICs can effectively accelerate collective communication primitives by moving computation into the network layer. The NetFPGA‑based offload of MPI_Scan yields significant latency and throughput improvements while reducing CPU overhead, highlighting a promising path for enhancing the performance of high‑performance computing applications that rely heavily on collective operations.