改进带动量的符号优化方法分析，实现小批量稳定收敛。
Improved Analysis for Sign-based Methods with Momentum Updates
- 在标准光滑性条件下，用常数批量实现相同收敛速度。
- 理论提升达√d倍，分布式场景下收敛更快。
- 适合大规模分布式训练与低精度优化场景。
本文改进了带动量的符号优化算法的理论分析。传统符号方法在可分光滑性假设下收敛速率为$\mathcal{O}(T^{-1/4})$，但需大批次或对噪声有额外假设。本文证明，带动量的signSGD可在常数批量下达到相同速率，且无需额外假设。分析基于标准$l_2$-光滑性条件，相比先前方法提升$\mathcal{O}(d^{1/2})$倍，其中$d$为问题维度。此外，在分布式设置中引入多数投票机制，提出的方法在两种情形下分别达到$\mathcal{O}(d^{1/2}T^{-1/2} + dn^{-1/2})$和$\mathcal{O}(\max \{ d^{1/4}T^{-1/4}, d^{1/10}T^{-1/5} \})$的收敛率，优于此前$\mathcal{O}(dT^{-1/4} + dn^{-1/2})$和$\mathcal{O}(d^{3/8}T^{-1/8})$的结果。数值实验验证了方法有效性。
原文摘要 · Abstract (English)
In this paper, we present enhanced analysis for sign-based optimization algorithms with momentum updates. Traditional sign-based methods, under the separable smoothness assumption, guarantee a convergence rate of $\mathcal{O}(T^{-1/4})$, but they either require large batch sizes or assume unimodal symmetric stochastic noise. To address these limitations, we demonstrate that signSGD with momentum can achieve the same convergence rate using constant batch sizes without additional assumptions. Our analysis, under the standard $l_2$-smoothness condition, improves upon the result of the prior momentum-based signSGD method by a factor of $\mathcal{O}(d^{1/2})$, where $d$ is the problem dimension. Furthermore, we explore sign-based methods with majority vote in distributed settings and show that the proposed momentum-based method yields convergence rates of $\mathcal{O}\left( d^{1/2}T^{-1/2} + dn^{-1/2} \right)$ and $\mathcal{O}\left( \max \{ d^{1/4}T^{-1/4}, d^{1/10}T^{-1/5} \} \right)$, which outperform the previous results of $\mathcal{O}\left( dT^{-1/4} + dn^{-1/2} \right)$ and $\mathcal{O}\left( d^{3/8}T^{-1/8} \right)$, respectively. Numerical experiments further validate the effectiveness of the proposed methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。