提出新型随机谱预条件优化方法，有效解决非凸问题的收敛难题。
Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives
- 基于谱梯度设计带约束的近端预条件算法，支持多种凸与非凸约束。
- 在重尾噪声下实现收敛，且方差缩减版本在标准噪声下更快收敛。
- 揭示多项式迭代更适配非线性预条件器，贴近实际优化器实现。
本文提出一类近端预条件梯度方法，聚焦于谱梯度框架，扩展了Muon和Scion优化器。设计了一类可处理广泛凸与非凸约束的随机算法，并通过针对所提方法几何结构的新分析，证明其在重尾噪声下的收敛性。进一步提出方差缩减版本，在标准噪声假设下实现更快收敛。最后表明，Muon中使用的多项式迭代更宜由非线性预条件器刻画，而非理想矩阵符号函数，从而建立更贴近实际实现的收敛分析。
原文摘要 · Abstract (English)
In this work, we develop proximal preconditioned gradient methods with a focus on spectral gradient methods providing a proximal extension to the Muon and Scion optimizers. We introduce a family of stochastic algorithms that can handle a wide variety of convex and nonconvex constraints and study its convergence under heavy-tailed noise, through a novel analysis tailored to the geometry of the proposed methods. We further propose a variance-reduced version, which achieves faster convergence under standard noise assumptions. Finally, we show that the polynomial iterations used in Muon are more accurately captured by a nonlinear preconditioner than by the ideal matrix sign, leading to a convergence analysis that more faithfully reflects practical implementations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。