通过计算通信重叠，显著加速去中心化训练。
Accelerating Decentralized Optimization via Overlapping Local Steps
- 设计新更新机制，实现计算与通信并行。
- 理论证明迭代复杂度不变，但每轮运行更快。
- 无需改造框架，适合实际部署的去中心化学习。
去中心化优化已成为分布式学习的关键范式，通过节点间直接协作实现可扩展训练并保护数据隐私。然而，现有方法常因节点频繁同步导致通信瓶颈。本文提出重叠局部去中心化SGD（OLDSGD），通过计算-通信重叠显著减少网络空闲时间。其精心设计的更新方式在保持与局部SGD相同平均更新量的同时，避免了通信引起的停滞。理论上，我们建立了平滑非凸目标下的非渐近收敛率，表明OLDSGD与标准去中心化SGD具有相同的迭代复杂度，但每轮运行时间更短。实验结果表明，在不同通信延迟下，OLDSGD均能持续提升实际运行时间的收敛速度。仅需对现有框架进行微小修改，即可提供一种无需牺牲理论保证的高效去中心化学习方案。
原文摘要 · Abstract (English)
Decentralized optimization has emerged as a critical paradigm for distributed learning, enabling scalable training while preserving data privacy through peer-to-peer collaboration. However, existing methods often suffer from communication bottlenecks due to frequent synchronization between nodes. We present Overlapping Local Decentralized SGD (OLDSGD), a novel approach to accelerate decentralized training by computation-communication overlapping, significantly reducing network idle time. With a deliberately designed update, OLDSGD preserves the same average update as Local SGD while avoiding communication-induced stalls. Theoretically, we establish non-asymptotic convergence rates for smooth non-convex objectives, showing that OLDSGD retains the same iteration complexity as standard Local Decentralized SGD while improving per-iteration runtime. Empirical results demonstrate OLDSGD's consistent improvements in wall-clock time convergence under different levels of communication delays. With minimal modifications to existing frameworks, OLDSGD offers a practical solution for faster decentralized learning without sacrificing theoretical guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。