Controlled disagreement improves generalization in decentralized training

Decentralized training is often regarded as inferior to centralized training because the consensus errors between workers are thought to undermine convergence and generalization, even with homogeneous data distributions. This work challenges this view by introducing decentralized SGD with Adaptive Consensus (DSGD-AC), which intentionally preserves non-vanishing consensus errors through a time-dependent scaling mechanism. We prove that these errors are not random noise but systematically align with the dominant Hessian subspace, acting as structured perturbations that guide optimization toward flatter minima. Across image classification and machine translation benchmarks, DSGD-AC consistently surpasses both standard DSGD and centralized SGD in test accuracy and solution flatness. Together, these results establish consensus errors as a useful implicit regularizer and open a new perspective on the design of decentralized learning algorithms.

💡 Research Summary

**
The paper challenges the prevailing belief that consensus errors in decentralized stochastic gradient descent (DSGD) are detrimental to convergence and generalization. It introduces a novel algorithm, Decentralized SGD with Adaptive Consensus (DSGD‑AC), which deliberately preserves non‑vanishing consensus errors by scaling the consensus regularization term with a time‑dependent factor γ(t). The authors prove that, contrary to being random noise, these errors systematically align with the dominant subspace of the Hessian matrix, acting as structured curvature‑aware perturbations that steer optimization toward flatter minima—an effect known to improve generalization.

The authors first observe that in standard DSGD the consensus error ‖e_i(t)‖ shrinks to zero as the learning rate α(t) decays, because the consensus regularizer becomes dominant in the surrogate objective J(t). This vanishing eliminates the “sharpness” term that could otherwise provide an implicit regularization effect. To retain a useful perturbation, DSGD‑AC multiplies the consensus regularizer by γ(t)=g₀·α(t)^{p}, where p≥2. Under a cosine annealing schedule, this choice yields a disagreement radius r_t²≈Θ(α(t)²/γ(t)) that stays at a constant order when p=2 and even grows mildly toward the end of training when p>2.

Mathematically, the paper rewrites the update in matrix form and projects the consensus error onto the eigenbasis of the graph Laplacian L=I−W. In this basis, the dynamics of each mode k follow Z_k(t)=Z_k(t‑1)(1−γ(t)λ_k)−α(t)Ĝ_k(t‑1), where λ_k are Laplacian eigenvalues. Because γ(t)·λ_k attenuates high‑frequency modes (large λ_k) while preserving low‑frequency ones, the remaining error concentrates along directions associated with large Hessian eigenvalues of the loss. The authors formalize this with Proposition 3.1, showing that a scaling exponent p≥2 is necessary to keep the disagreement radius non‑trivial as α(t)→0.

Empirically, the authors evaluate DSGD‑AC on several benchmarks: CIFAR‑10/100 with Wide ResNet‑28‑10, ImageNet with ResNet‑50, and WMT14 English‑German translation with a Transformer‑Base. Experiments use 8–16 workers arranged in a one‑peer ring topology and i.i.d. data partitions. DSGD‑AC consistently outperforms vanilla DSGD and centrally‑trained SGD (optimally tuned) in test accuracy (e.g., +0.8 % on CIFAR‑10, +1.2 % on CIFAR‑100) and exhibits flatter minima, as measured by reduced top Hessian eigenvalues and lower loss sensitivity to random perturbations. The method incurs negligible extra computation (<2 % overhead) and no additional communication beyond standard DSGD.

A sensitivity analysis shows that p<2 leads to rapid decay of consensus errors and degraded performance, while p>2 can cause excessive disagreement and instability; p≈3 with g₀≈0.5–0.8 yields the best trade‑off. The authors also discuss limitations: all experiments assume i.i.d. data across workers, and the behavior under heterogeneous (non‑i.i.d.) data remains an open question. Moreover, the analysis relies on spectral properties of the Laplacian; more complex or dynamic topologies may require refined scaling strategies.

In conclusion, the paper establishes that consensus errors, when properly controlled, serve as a free, curvature‑aware regularizer that improves generalization without sacrificing the communication efficiency of decentralized training. This insight opens a new design space for decentralized learning algorithms, suggesting that “consensus errors should be harnessed rather than eliminated.”