FCDP: Fully Cached Data Parallel for Communication-Avoiding Large-Scale Training

Training billion-parameter models requires distributing model states across GPUs using fully sharded data parallel (i.e., ZeRO-3). While ZeRO-3 succeeds on clusters with high-bandwidth NVLink and InfiniBand interconnects, researchers with commodity hardware face severe inter-node all-gather bottlenecks. Existing optimizations take two approaches: GPU memory caching (MiCS, ZeRO++) trades memory capacity for reduced communication, triggering out-of-memory failures on large models; host memory offloading (ZeRO-Offload, ZeRO-Infinity) extends capacity but degrades throughput due to PCIe overhead. We observe that on bandwidth-limited clusters, host memory can serve not as an overflow tier but as a fast caching layer that outperforms inter-node communication. Based on this insight, we propose FCDP, which eliminates redundant inter-node communication while preserving ZeRO-3’s minimal GPU memory footprint. FCDP caches forward-pass parameters in host memory and reuses them during the backward pass via fast intra-node all-gather, reducing inter-node all-gather by 50%. For parameter-efficient fine-tuning (PEFT), FCDP selectively communicates only trainable parameters to maximize caching, reducing inter-node traffic by over 99%. In our commodity cluster setup, FCDP achieves up to 100x higher throughput than ZeRO-3 and 51x higher than ZeRO++, while maintaining ZeRO-3’s maximum batch size.

💡 Research Summary

The paper addresses a critical bottleneck in large‑scale model training on commodity clusters: the inter‑node all‑gather operations required by ZeRO‑3 (full sharding) become prohibitively expensive when network bandwidth is limited. While ZeRO‑3 minimizes GPU memory usage by sharding parameters, gradients, and optimizer states across all devices, it must reconstruct the full parameter tensor twice per layer (once for the forward pass and once for the backward pass) via inter‑node all‑gather, followed by a reduce‑scatter for gradients. On high‑performance DGX‑style systems with NVLink/NVSwitch and 100‑Gbps InfiniBand, this cost is tolerable, but on typical research‑grade or cloud environments where GPUs are connected via PCIe and only a single NIC per node is available, the inter‑node bandwidth can be an order of magnitude lower than intra‑node links. Consequently, ZeRO‑3’s communication dominates training time, leading to up to a 5.9× slowdown as network quality degrades.

Existing mitigations fall into two categories. GPU‑memory caching approaches (MiCS, ZeRO++) reduce communication by keeping a copy of parameters on each GPU, but they increase per‑GPU memory consumption, causing out‑of‑memory failures for very large models. Host‑memory offloading techniques (ZeRO‑Offload, ZeRO‑Infinity) extend capacity but incur additional PCIe transfers for every access, which hurts throughput. Both families ignore the fact that, on bandwidth‑limited clusters, a PCIe transfer from host to GPU can be faster than an inter‑node network transfer.

The authors’ key insight (C1) is that host memory can serve as a fast, local cache rather than merely an overflow tier. Empirically, moving 16 GB of data over PCIe 4.0 takes ~0.61 s, while a 100 Gbps RDMA all‑gather takes ~0.95 s, and slower Ethernet links are even worse. This motivates the design of Fully Cached Data Parallel (FCDP).

FCDP’s core mechanisms are:

1. Forward‑pass caching – After the forward‑pass all‑gather, each GPU writes its received shard to host memory.
2. Backward‑pass reuse – During the backward pass, the GPU reads the cached shard from host memory and performs an intra‑node all‑gather (PCIe/NVLink) to reconstruct the full tensor locally. This eliminates the inter‑node all‑gather for the backward pass, cutting inter‑node traffic by 50 % (C2).
3. Dynamic memory pressure adaptation – GPU memory availability is monitored at runtime. When spare GPU memory exists, parameters are cached on‑device to avoid PCIe latency; under pressure they are moved to host memory. This guarantees that the worst‑case memory footprint matches ZeRO‑3 while opportunistically reducing PCIe transfers (C3).
4. PEFT‑aware communication – For parameter‑efficient fine‑tuning (e.g., LoRA), parameters are classified as trainable or frozen at initialization. Frozen weights are gathered once, cached indefinitely in host memory, and never communicated again. Only the small trainable subset participates in inter‑node all‑gather, reducing communication by >99 % when >99 % of parameters are frozen (C4).

The system is implemented on top of PyTorch’s ZeRO‑3 codebase, adding a host‑cache manager, a scheduler that overlaps PCIe copies with compute, and a PEFT‑aware collective engine. The authors evaluate FCDP on a 4‑node (32‑GPU) commodity cluster with various network configurations (100 Gbps RDMA, IPoIB, 10 Gbps Ethernet). Experiments cover full fine‑tuning of GPT‑style models (10 B–30 B parameters) and LoRA fine‑tuning.

Key results:

• Full fine‑tuning – FCDP achieves up to 41.3 % higher throughput than ZeRO‑3 and roughly 2× higher than ZeRO++. The maximum batch size remains identical to ZeRO‑3, confirming that GPU memory usage is unchanged.
• LoRA fine‑tuning – By eliminating inter‑node communication for frozen weights, FCDP reaches up to 100× the throughput of ZeRO‑3 and 51× that of ZeRO++.
• Scalability – As the number of nodes grows, the relative advantage of FCDP increases because the inter‑node traffic saved per layer becomes a larger fraction of total runtime.

Table I in the paper summarizes the trade‑offs across systems, highlighting that FCDP uniquely combines “GPU memory minimal”, “inter‑node communication reduced”, and “PEFT‑aware” properties. Theoretical analysis in Section VI quantifies the bandwidth gap (intra‑node 2.5–16× faster than inter‑node) and shows how the cached‑parameter model reduces the number of collective calls from three per layer to two (forward all‑gather + intra‑node gather for backward).

In conclusion, FCDP demonstrates that host memory, when used as an active caching tier, can dramatically alleviate inter‑node communication bottlenecks without sacrificing the memory efficiency of full sharding. This enables researchers with modest hardware to train billion‑parameter models and perform PEFT at speeds previously attainable only on expensive DGX‑style clusters. Future work may explore integration with emerging PCIe‑based peer‑to‑peer memory, support for other PEFT methods (e.g., prompt tuning), and extending the dynamic caching policy to multi‑tenant environments.