提出无需调参的误差反馈机制,提升分布式训练收敛速度。
Improved Convergence in Parameter-Agnostic Error Feedback through Momentum
- 结合归一化更新与动量,使用随时间递减步长,避免依赖问题参数。
- 理论证明收敛速率接近最优,如重球法达$O(1/T^{1/4})$。
- 适合大规模神经网络训练,实验证实理论有效性。
通信压缩对大规模机器学习模型的分布式训练至关重要,但会因引入噪声而影响收敛性。误差反馈(EF)机制被广泛用于缓解分布式压缩算法的这一问题。然而,现有分布式EF算法通常需要预先知晓问题参数(如光滑常数)以调整步长,限制了其在大规模神经网络训练中的实用性。本文研究将误差反馈与归一化更新、多种动量变体及参数无关的时间可变步长相结合的算法,从而消除对问题依赖调参的需求。我们分析了这些算法在最小化平滑函数时的收敛性,建立了接近最优的参数无关复杂度界。具体而言,归一化EF21在Polyak重球动量下达到近$O(1/T^{1/4})$的收敛速率,在IGT下为$O(1/T^{2/7})$,在STORM和海森修正动量下为$O(1/T^{1/3})$。结果在递减步长和小批量情况下仍成立。实验验证了理论结论。
原文摘要 · Abstract (English)
Communication compression is essential for scalable distributed training of modern machine learning models, but it often degrades convergence due to the noise it introduces. Error Feedback (EF) mechanisms are widely adopted to mitigate this issue of distributed compression algorithms. Despite their popularity and training efficiency, existing distributed EF algorithms often require prior knowledge of problem parameters (e.g., smoothness constants) to fine-tune stepsizes. This limits their practical applicability especially in large-scale neural network training. In this paper, we study normalized error feedback algorithms that combine EF with normalized updates, various momentum variants, and parameter-agnostic, time-varying stepsizes, thus eliminating the need for problem-dependent tuning. We analyze the convergence of these algorithms for minimizing smooth functions, and establish parameter-agnostic complexity bounds that are close to the best-known bounds with carefully-tuned problem-dependent stepsizes. Specifically, we show that normalized EF21 achieve the convergence rate of near ${O}(1/T^{1/4})$ for Polyak's heavy-ball momentum, ${O}(1/T^{2/7})$ for Iterative Gradient Transport (IGT), and ${O}(1/T^{1/3})$ for STORM and Hessian-corrected momentum. Our results hold with decreasing stepsizes and small mini-batches. Finally, our empirical experiments confirm our theoretical insights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。