针对带重尾噪声的去中心化非凸优化，提出高效算法并证明近最优复杂度。
Near-Optimal Decentralized Stochastic Nonconvex Optimization with Heavy-Tailed Noise
- 设计带梯度追踪的归一化随机梯度下降算法
- 在重尾噪声下达到最优样本复杂度和近最优通信复杂度
- 适用于分布式机器学习中噪声鲁棒性要求高的场景
本文研究行随机网络下的去中心化随机非凸优化问题。考虑实际应用中普遍存在的重尾梯度噪声，提出一种去中心化归一化随机梯度下降结合Pull-Diag梯度追踪的方法，在保证逼近驻点的同时，实现了最优样本复杂度与近最优通信复杂度。进一步在无向网络设置下沿用该框架，也达到了近乎紧致的上界复杂度。实验验证了所提方法在实际任务中的优越性能。
原文摘要 · Abstract (English)
This paper studies decentralized stochastic nonconvex optimization problem over row-stochastic networks. We consider the heavy-tailed gradient noise which is empirically observed in many popular real-world applications. Specifically, we propose a decentralized normalized stochastic gradient descent with Pull-Diag gradient tracking, which achieves approximate stationary points with the optimal sample complexity and the near-optimal communication complexity. We further follow our framework to study the setting of undirected networks, also achieving the nearly tight upper complexity bounds. Moreover, we conduct empirical studies to show the practical superiority of the proposed methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。