Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems

Deploying large-scale MoE models presents challenges in memory capacity and bandwidth for expert activation. While Attention-FFN Disaggregation (AFD) has emerged as a potential architecture to decouple compute and memory resources, its performance boundaries compared to standard large-scale Expert Parallelism (EP) remain underexplored. In this paper, we conduct a systematic analysis of AFD by extending the roofline model to the communication level, correlating interconnect bandwidth, arithmetic intensity, and Hardware FLOPS Utilization (HFU). Our analysis reveals a dead zone on standard clusters: increasing FFN instance count fails to improve HFU as computational workload is capped by scale-out bandwidth, causing operator active time to shrink relative to the fixed latency budget. We further show that AFD’s discrete node-level scaling incurs higher imbalance penalties than EP’s continuous batch adjustment. Nevertheless, these limitations diminish under specific conditions: Superpod-class hardware with abundant interconnect bandwidth and models with coarse-grained experts and lower sparsity are more likely to benefit from AFD. These findings position AFD as a promising approach for specific hardware-model combinations rather than a universal solution.

💡 Research Summary

The paper conducts a comprehensive investigation of Attention‑FFN Disaggregation (AFD), a recently proposed architecture for large‑scale Mixture‑of‑Experts (MoE) language models, and compares it against the more established Expert Parallelism (EP) approach. The authors begin by outlining the challenges faced when deploying trillion‑parameter models such as DeepSeek‑V3 and Kimi‑K2: massive parameter counts, large KV caches, and the need to split inference across dozens of GPUs. While pre‑fill/ decode disaggregation (PD) has already alleviated the attention bottleneck during the pre‑fill stage, the decode stage remains memory‑bandwidth and communication‑bound because only a small fraction of experts are active per token.

AFD tackles this by physically separating the attention pipeline from the FFN (expert) pipeline, assigning them to distinct sets of nodes (A‑role and F‑role). Tokens are dispatched from attention nodes to FFN nodes according to gating results, and later combined after the expert computation. This creates a unidirectional M→N collective communication pattern, in contrast to EP’s symmetric all‑to‑all. The authors extend the classic Roofline performance model to include this communication level, linking arithmetic intensity (I), inter‑connect bandwidth (B), and Hardware FLOPS Utilization (HFU). They also introduce Operator FLOPS Utilization (OFU) and temporal sparsity (S_t) to capture the fraction of the allocated latency budget (t_B) actually spent on computation.

A key contribution is the identification of a “dead zone” on conventional clusters (e.g., NVIDIA H800‑class). In this region, scaling the number of FFN instances (scale‑out) does not increase HFU because the dispatch/combine latency (t_c) dominates the fixed per‑micro‑batch budget t_B. Even when FFN parallelism is increased, the arithmetic intensity of each expert (I ≈ 2·B, where B is the average tokens per expert) remains low, so the system stays network‑bound. Consequently, HFU plateaus around 30‑40 % and the expected throughput gains of AFD never materialize.

The paper further shows that AFD requires at least three‑batch overlap (3BO) to avoid pipeline bubbles. With two‑batch overlap (2BO) or no overlap, the total latency of dispatch + FFN + combine exceeds the attention latency (t_a), causing idle periods on the attention side that quickly propagate back and forth, leading to severe performance collapse. 3BO can hide communication latency but leaves a very narrow margin for latency jitter; any increase in t_c or imbalance in t_a/t_f immediately violates the budget.

Imbalance sensitivity is another critical finding. Because A‑role and F‑role nodes are scaled independently, any disparity in token distribution (DP imbalance) or expert load (EP imbalance) directly inflates either t_a or t_f. EP, by contrast, can adjust batch sizes continuously across all ranks, mitigating imbalance effects. Experiments demonstrate that under identical imbalance levels, AFD suffers a larger drop in HFU and experiences more frequent bubbles than EP.

Despite these drawbacks, the authors identify scenarios where AFD shines. On “Superpod‑class” hardware equipped with ultra‑high‑bandwidth interconnects (InfiniBand HDR, > 600 GB/s), the dispatch/combine latency becomes small enough that t_a ≈ t_f ≈ t_B, allowing both pipelines to run near full utilization. Moreover, models with coarse‑grained experts (large hidden size H relative to MoE intermediate size M) and lower sparsity (Top‑K ratio 0.1‑0.2) increase the average tokens per expert B, thereby raising arithmetic intensity and OFU. In such configurations, AFD can achieve OFU above 70 % and overall HFU improvements of 10‑15 % over EP. The presence of Multi‑Token Prediction (MTP) further relaxes the latency budget by increasing the average acceptance length L_accept, which widens t_B and makes 3BO easier to satisfy.

The experimental section validates these claims across a matrix of hardware (standard GPU clusters vs. Superpod) and model parameters (expert counts 64‑256, sparsity 0.05‑0.3). On standard clusters, even quadrupling FFN instances yields HFU below 35 % and bubble rates above 20 %. On Superpod, the same scaling pushes HFU to ~78 % with bubble rates under 3 %, confirming the importance of network bandwidth. When compared head‑to‑head with EP at equal token throughput, AFD outperforms EP only when the interconnect is sufficiently fast; otherwise it underperforms by up to 15 %.

In conclusion, the paper positions AFD not as a universal replacement for EP but as a hardware‑aware optimization that can be advantageous under specific conditions: (1) ultra‑high‑bandwidth, low‑latency interconnects; (2) coarse‑grained expert designs with relatively low sparsity; (3) availability of MTP to enlarge the latency budget; and (4) workloads that can sustain three‑batch overlap without excessive jitter. The authors provide a practical decision framework for practitioners: evaluate interconnect bandwidth, expert granularity, sparsity, and MTP support before committing to AFD. When these criteria are met, AFD offers a promising path to higher hardware utilization and reduced inference cost for next‑generation trillion‑parameter MoE models.