提出谱分解方法，让神经网络训练更高效地学习曲率矩阵。
Spectral-factorized Positive-definite Curvature Learning for NN Training
- 用谱分解动态适应正定曲率估计，避免昂贵的矩阵开方计算。
- 在无梯度优化中实现协方差自适应，提升收敛效率。
- 适用于多种神经网络训练场景，兼顾性能与计算效率。
许多训练方法（如 Adam(W)、Shampoo）通过学习正定曲率矩阵并应用其逆平方根进行预条件化。近期非对角方法（如 Shampoo）受到关注，但因需通过矩阵分解计算矩阵平方根，导致计算成本高，且仅适用于特定类型的曲率信息。为此，我们提出一种黎曼优化方法，动态适配谱分解的正定曲率估计，实现任意矩阵根的高效应用和通用曲率学习。我们在正定矩阵优化、无梯度优化中的协方差自适应以及神经网络训练的曲率学习中验证了该方法的有效性与通用性。
原文摘要 · Abstract (English)
Many training methods, such as Adam(W) and Shampoo, learn a positive-definite curvature matrix and apply an inverse root before preconditioning. Recently, non-diagonal training methods, such as Shampoo, have gained significant attention; however, they remain computationally inefficient and are limited to specific types of curvature information due to the costly matrix root computation via matrix decomposition. To address this, we propose a Riemannian optimization approach that dynamically adapts spectral-factorized positive-definite curvature estimates, enabling the efficient application of arbitrary matrix roots and generic curvature learning. We demonstrate the efficacy and versatility of our approach in positive-definite matrix optimization and covariance adaptation for gradient-free optimization, as well as its efficiency in curvature learning for neural net training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。