提出新型分布式优化算法,实现高概率收敛且无需强假设。
High-Probability Convergence in Decentralized Stochastic Optimization with Gradient Tracking

- 引入梯度追踪改进分布式随机梯度下降,降低对数据异质性的要求。
- 在非凸和PL条件下分别达到最优高概率收敛率,与均方误差结果一致。
- 首次在高概率意义下证明偏差修正技术的有效性,适合实际分布式训练场景。
本文研究分布式随机优化中的高概率(HP)收敛性,多个代理通过网络协同训练模型。现有高概率结果几乎仅针对需要强假设(如有界数据异质性或强凸代价)的分布式随机梯度下降(DSGD)。而均方误差(MSE)结果表明,加入偏差修正的技术可放宽条件并提升实际性能。本文首次填补此差距,分析引入梯度追踪的DSGD(GT-DSGD)在满足弱亚高斯噪声条件下的高概率收敛性。结果表明,GT-DSGD在非凸和Polyak-Łojasiewicz(PL)代价下分别达到阶为\(\mathcal{O}\left(\frac{\log(1/δ)}{\sqrt{nT}}\right)\)和\(\mathcal{O}\left(\frac{\log(1/δ)}{nT}\right)\)的最优高概率收敛率,其中\(n\)为代理数,\(T\)为时间步长,\(δ∈(0,1)\)为置信参数。理论表明,在与MSE相同条件下,GT-DSGD可实现高概率收敛,且瞬态性能相当。据我们所知,这是首个针对包含偏差修正的分布式优化方法的高概率保证。真实与合成数据上的实验验证了理论结果,凸显了GT-DSGD的优越性能。
原文摘要 · Abstract (English)
We study high-probability (HP) convergence guarantees in decentralized stochastic optimization, where multiple agents collaborate to jointly train a model over a network. Existing HP results in decentralized settings almost exclusively focus on the Decentralized Stochastic Gradient Descent ($\mathtt{DSGD}$) algorithm, which requires strong assumptions, such as bounded data heterogeneity, or strong convexity of each agent's cost. This is contrary to the mean-squared error (MSE) results, where methods incorporating bias-correction techniques are known to converge under relaxed assumptions and achieve better practical performance. In this paper we provide the first step toward bridging the gap, by studying HP convergence of $\mathtt{DSGD}$ incorporating the gradient tracking technique, in the presence of noise satisfying a relaxed sub-Gaussian condition. We show that the resulting method, dubbed $\mathtt{GT-DSGD}$, achieves order-optimal HP convergence rates for both non-convex and Polyak-Łojasiewicz costs, of order $\mathcal{O}\Big(\frac{\log(1/δ)}{\sqrt{nT}}\Big)$ and $\mathcal{O}\Big(\frac{\log(1/δ)}{nT}\Big)$, respectively, where $n$ is the number of agents, $T$ is the time horizon and $δ\in (0,1)$ is the confidence parameter. Our results establish that $\mathtt{GT-DSGD}$ converges in the HP sense under the same conditions on the cost as in the MSE sense, while achieving comparable transient times. To the best of our knowledge, these are the first HP guarantees for decentralized optimization methods incorporating bias-correction. Numerical experiments on real and synthetic data verify our theoretical findings, underlining the superior performance of $\mathtt{GT-DSGD}$ and highlighting that the benefits of incorporating bias-correction are also maintained in the HP sense.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。