arXiv:2604.09970cs.LGcs.DC2026-04中稿 · TMLR

LoDAdaC让去中心化训练更快更省通信,支持自适应优化和压缩。

LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication

  • 基于多轮本地训练与自适应梯度更新,兼容Adam等优化器
  • 通信成本降低倍数显著,收敛速度优于现有算法
  • 适合大规模分布式训练场景,尤其对通信受限系统友好

在去中心化分布式学习中,实现快速收敛与低通信开销对可扩展性和高效率至关重要。自适应梯度方法(如Adam)在深度学习和集中式设置中表现出色,但在包含多轮本地训练的去中心化场景(如联邦学习)中的收敛性尚未充分研究。为此,我们提出LoDAdaC——一种统一的多轮本地训练(MLT)去中心化框架,结合Adam型更新与压缩通信(CC)。LoDAdaC支持多种本地自适应优化器,包括AMSGrad、Adam和AdaGrad;兼容标准(可能有偏)压缩技术,如低比特量化和稀疏化。通过多轮本地训练与压缩通信,实现了通信成本的多重降低;自适应更新机制则加速了收敛。我们通过复杂度分析严格证明了该组合优势。图像分类与GPT风格语言模型训练实验验证了理论结果,表明LoDAdaC在收敛速度和通信效率上显著优于现有去中心化算法。

原文摘要 · Abstract (English)

In the decentralized distributed learning, achieving fast convergence and low communication cost is essential for scalability and high efficiency. Adaptive gradient methods, such as Adam, have demonstrated strong practical performance in deep learning and centralized distributed settings. However, their convergence properties remain largely unexplored in decentralized settings involving multiple local training steps, such as federated learning. To address this limitation, we propose LoDAdaC, a unified multiple Local Training (MLT) Decentralized framework with Adam-type updates and Compressed communication (CC). LoDAdaC accommodates a broad class of optimizers for its local adaptive updates, including AMSGrad, Adam, and AdaGrad; it is compatible with standard (possibly biased) compressors such as low-bit quantization and sparsification. MLT and CC enable LoDAdaC to achieve multiplied reduction of communication cost, while the technique of adaptive updates enables fast convergence. We rigorously prove the combined advantage through complexity analysis. In addition, experiments on image classification and GPT-style language model training validate our theoretical findings and show that LoDAdaC significantly outperforms existing decentralized algorithms in terms of convergence speed and communication efficiency.

去中心化学习自适应优化通信压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。