提出IGT加速LMO优化，单次梯度迭代即可实现更优收敛速度。
Accelerating LMO-Based Optimization via Implicit Gradient Transport
- 利用隐式梯度传输，在不增加梯度计算的前提下改进收敛性。
- 理论证明：仅用一次随机梯度，迭代复杂度达O(ε⁻³.⁵)，优于传统方法。
- 适用于追求高效优化的深度学习场景，尤其适合现代LMO类算法改进。
近期优化器如Lion和Muon通过线性最小化算子（LMO）归一化梯度动量，展现出优异的实验性能。尽管已有研究尝试通过方差减少加速LMO方法，但通常需额外梯度评估，带来显著计算开销。同时，对LMO方法的理论理解在无约束与约束形式间仍不统一。为此，本文提出新型随机LMO方法LMO-IGT，基于隐式梯度传输（IGT）。我们进一步构建了统一的随机LMO优化框架，并引入新的平稳性度量——正则化支撑函数（RSF），将梯度范数与Frank-Wolfe间隙统一于同一框架。通过在传输点评估随机梯度，LMO-IGT在保持单梯度/迭代结构的同时实现加速。分析表明，随机LMO迭代复杂度为O(ε⁻⁴)，方差减少型LMO为O(ε⁻³)（需额外梯度），而LMO-IGT仅用一次随机梯度即达O(ε⁻³.⁵)。实验显示，LMO-IGT持续优于标准随机LMO，且开销极小。其中，Muon-IGT在多种设置下表现最优，验证了IGT是现代LMO优化的有效且实用的加速机制。
原文摘要 · Abstract (English)
Recent optimizers such as Lion and Muon have demonstrated strong empirical performance by normalizing gradient momentum via linear minimization oracles (LMOs). While variance reduction has been explored to accelerate LMO-based methods, it typically incurs substantial computational overhead due to additional gradient evaluations. At the same time, the theoretical understanding of LMO-based methods remains fragmented across unconstrained and constrained formulations. Motivated by these limitations, we propose \emph{LMO-IGT}, a new class of stochastic LMO-based methods leveraging implicit gradient transport (IGT). We further introduce a unified framework for stochastic LMO-based optimization together with a new stationarity measure, the \emph{regularized support function} (RSF), which bridges gradient-norm and Frank--Wolfe-gap notions within a common framework. By evaluating stochastic gradients at transported points, LMO-IGT accelerates convergence while retaining the single-gradient-per-iteration structure of standard stochastic LMO. Our analysis establishes that stochastic LMO achieves an iteration complexity of $\mathcal{O}(\varepsilon^{-4})$, variance-reduced LMO achieves $\mathcal{O}(\varepsilon^{-3})$ at the cost of additional gradient evaluations, and LMO-IGT achieves $\mathcal{O}(\varepsilon^{-3.5})$ using only a single stochastic gradient per iteration. Empirically, LMO-IGT consistently improves over stochastic LMO counterparts with negligible overhead. Among its instantiations, Muon-IGT achieves the strongest overall performance across evaluated settings, demonstrating that IGT provides an effective and practical acceleration mechanism for modern LMO-based optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。