不用迭代就能快速正交化，提速且稳定
IFNSO: Iteration-Free Newton-Schulz Orthogonalization
- 把传统迭代方法改成一次性公式，省去重复矩阵运算
- 在多个任务上比现有方法更快更稳，计算效率显著提升
- 适合需要高效正交优化的深度学习场景
牛顿-舒尔茨（NS）迭代已成为穆昂等优化器及斯特夫尔流形优化中的关键正交化技术。尽管效果良好，传统NS迭代因反复进行高维矩阵乘法而带来显著计算开销。为克服此局限，我们提出无迭代牛顿-舒尔茨正交化（IFNSO），将传统迭代结构整合为统一的无迭代形式。通过分析各矩阵幂次的贡献，剔除不重要项，并引入可学习系数的多项式，这些系数经优化以确保卓越的计算效率与稳定收敛。大量实验表明，IFNSO在性能上优于现有方法。代码已开源：https://github.com/greekinRoma/Unified_Newton_Schulz_Orthogonalization。
原文摘要 · Abstract (English)
The Newton-Schulz (NS) iteration has become a key technique for orthogonalization in optimizers such as Muon and for optimization on the Stiefel manifold. Despite its effectiveness, the conventional NS iteration incurs significant computational overhead due to repeated high-dimensional matrix multiplications. To overcome these limitations, we propose Iteration-Free Newton-Schulz Orthogonalization (IFNSO), a novel framework that consolidates the traditional iterative structure into a unified and Iteration-Free formulation. By analyzing the contribution of individual matrix powers, we streamline the process by removing insignificant terms and introducing a polynomial with learnable coefficients. These coefficients are optimized to ensure both superior computational efficiency and stable convergence. Extensive experiments demonstrate that IFNSO achieves superior performance compared to existing methods. Our code is available at: https://github.com/greekinRoma/Unified_Newton_Schulz_Orthogonalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。