In-Network Collective Operations: Game Changer or Challenge for AI Workloads?

This paper summarizes the opportunities of in-network collective operations (INC) for accelerated collective operations in AI workloads. We provide sufficient detail to make this important field accessible to non-experts in AI or networking, fostering a connection between these communities. Consider two types of INC: Edge-INC, where the system is implemented at the node level, and Core-INC, where the system is embedded within network switches. We outline the potential performance benefits as well as six key obstacles in the context of both Edge-INC and Core-INC that may hinder their adoption. Finally, we present a set of predictions for the future development and application of INC.

💡 Research Summary

The paper provides a comprehensive overview of In‑Network Collective operations (INC) and evaluates their potential to transform large‑scale AI workloads, especially the training and inference of massive language models. It begins by describing the three dominant parallelism strategies used in modern deep‑learning systems—Data Parallelism (DP), Pipeline Parallelism (PP), and Tensor Parallelism (TP)—and explains how each relies heavily on collective communication primitives such as Allreduce, Allgather, Reduce_scatter, Broadcast, and Alltoall. Traditional implementations of these collectives run on CPUs or GPUs and require data to be copied to and from host memory, incurring significant latency and bandwidth overhead.

To address these inefficiencies, the authors distinguish two architectural categories of INC: Edge‑INC and Core‑INC. Edge‑INC moves the collective operation into the network interface (NI) of each node. Techniques such as Portals 4 and sPIN enable the NIC to stream data directly between accelerators without staging it in DRAM, thereby eliminating host memory traffic, reducing latency by tens of microseconds, and allowing full overlap of communication and computation. Edge‑INC also supports asynchronous progression and can implement advanced protocols like constant‑time multicast broadcast.

Core‑INC embeds simple arithmetic units inside the network switches themselves. By performing reductions (e.g., summations) inside the switch fabric, Core‑INC reduces the total amount of data that must traverse the network. For operations that involve reduction—Allreduce, Reduce_scatter, and Broadcast—Core‑INC can cut the required bandwidth roughly in half compared to endpoint‑only algorithms, because each packet is reduced once in the core and then broadcast back, rather than being sent twice (once for reduction, once for broadcast). The paper illustrates this with a tree‑based Allreduce that uses a single reduction tree in the switch layer versus a ring algorithm that requires two full passes. Operations that do not involve reduction, such as Alltoall, are less amenable to Core‑INC; they may still benefit from coordinated scheduling but require more sophisticated congestion‑avoidance mechanisms.

Both approaches have complementary strengths. Edge‑INC is resilient to switch failures and can fall back to traditional routing, but its performance is limited by the bandwidth of the node‑level links. Core‑INC offers higher scalability in the network core but introduces new failure modes: the state held in switches is not as easily recoverable, and a single switch failure can affect many collective operations.

The authors identify six major obstacles that must be overcome for widespread adoption of INC: (1) low‑precision data types (e.g., 4‑bit, 8‑bit) introduce overflow, underflow, and rounding errors that jeopardize numerical stability; (2) limited compute resources in switches make it difficult to support complex collectives like Alltoall; (3) fault tolerance and state recovery mechanisms for in‑switch computation are under‑developed; (4) the programming model requires new APIs, compiler support, and developer tooling, raising the barrier to entry; (5) security and privacy concerns arise when data is processed inside the network fabric; and (6) commercial switch firmware update cycles are slow, hindering rapid deployment of new INC features.

To mitigate these challenges, the paper proposes several research directions: dynamic scaling and mixed‑precision schemes to preserve accuracy while exploiting low‑bit formats; checkpointing and redundancy mechanisms inside switches; a standardized INC API layer that can be exposed through existing collective communication libraries (e.g., NCCL, MPI); hardware‑software co‑design that integrates cryptographic primitives for secure in‑network computation; and an open‑source firmware ecosystem to accelerate innovation.

Finally, the authors forecast that within the next five years AI‑specific smart switches and programmable NICs will become commonplace in large‑scale clusters. INC is expected to evolve from an experimental acceleration technique into a foundational component of distributed AI systems, delivering up to 2× traffic reduction for key collectives, lowering host memory pressure, and enabling tighter compute‑communication overlap. In multi‑tenant and multi‑cloud environments, these gains translate into lower operational costs and improved energy efficiency, positioning INC as a critical enabler for the next generation of AI infrastructure.