提出自适应阻尼方法，解决大模型优化中计算效率与精度的矛盾。
FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
- 根据误差动态调整阻尼因子和分解频率，实现稳定训练。
- 实验显示相比标准Shampoo，耗时更短且收敛稳定。
- 适合追求高效高精度训练的大规模模型优化场景。
Shampoo 在大规模优化基准上表现优异，但其矩阵求逆带来的计算开销极大；实践中常采用过时的预条件更新以缓解此问题，导致计算效率与优化保真度之间的权衡。本文从收敛性与稳定性双重视角分析了过时性（staleness）的影响：过时虽提升效率，却降低性能并引入数值不稳定性。关键发现是，阻尼作为数值稳定器可有效抑制这些负面影响。基于此，我们提出FOAM，一种自适应算法，通过近似过时误差动态控制阻尼因子与特征分解频率。实验表明，相比标准Shampoo，FOAM在减少实际运行时间的同时保持了鲁棒收敛性。
原文摘要 · Abstract (English)
Shampoo is attracting considerable attention for its superior performance on large-scale optimization benchmarks; yet it faces a significant practical bottleneck: the prohibitive computational overhead of matrix inversion. To mitigate this, practitioners typically rely on stale preconditioner updates, creating a fundamental trade-off between computational efficiency and optimization fidelity. In this work, we provide a theoretical study of staleness through the complementary lenses of convergence and stability. While staleness improves computational efficiency, it inherently degrades performance and introduces numerical instability. Crucially, we identify that damping, acting as a numerical stabilizer, can effectively suppress these negative effects. Guided by this analysis, we propose FOAM, an adaptive algorithm that stabilizes training by dynamically controlling both the damping factor and the eigendecomposition frequency based on an approximation of the staleness-oriented error. Experimental results demonstrate that FOAM reduces wall-clock time compared to standard Shampoo while maintaining robust convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。