首个在重尾噪声下有理论保证的非凸分布式双层优化算法。
Nonconvex Decentralized Stochastic Bilevel Optimization under Heavy-Tailed Noise
- 提出无截断的归一化方差缩减梯度下降法
- 首次建立重尾噪声下的收敛速率理论
- 适合处理语言模型等真实场景的噪声数据
现有分布式随机优化方法通常假设低层损失函数强凸且梯度噪声方差有限，但这些强假设在真实机器学习场景中往往不成立。例如，语言数据学习常导致重尾梯度。为此，本文针对非凸双层优化问题，在重尾噪声条件下提出一种新型分布式随机双层优化算法。具体而言，设计了一种无需任何截断操作的归一化随机方差缩减双层梯度下降算法，并通过创新性地控制重尾噪声下相互依赖的梯度序列，首次建立了该问题的收敛速率理论。据我们所知，这是首个在重尾噪声下具有严格理论保障的分布式双层优化算法。大量实验结果验证了该算法在处理重尾噪声方面的有效性。
原文摘要 · Abstract (English)
Existing decentralized stochastic optimization methods assume the lower-level loss function is strongly convex and the stochastic gradient noise has finite variance. These strong assumptions typically are not satisfied in real-world machine learning models. For example, learning on language data typically leads to heavy-tailed gradient. To address these limitations, we develop a novel decentralized stochastic bilevel optimization algorithm for the nonconvex bilevel optimization problem under heavy-tailed noise. Specifically, we develop a normalized stochastic variance-reduced bilevel gradient descent algorithm, which does not rely on any clipping operation. Moreover, we establish its convergence rate by innovatively bounding interdependent gradient sequences under heavy-tailed noise for nonconvex decentralized bilevel optimization problems. As far as we know, this is the first decentralized bilevel optimization algorithm with rigorous theoretical guarantees under heavy-tailed noise. The extensive experimental results confirm the effectiveness of our algorithm in handling heavy-tailed noise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。