用随机线性代数加速物理信息神经网络的自然梯度优化
Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra
- 引入随机数值线性代数技术预处理内层共轭梯度求解器
- 在多个偏微分方程问题上显著提升训练速度与收敛性
- 适合需要高效训练神经网络型微分方程求解器的研究者
自然梯度下降（NGD）作为训练基于神经网络的微分方程求解器（如物理信息神经网络，PINNs）的有前景优化算法，其实际应用常受限于涉及格拉姆矩阵的线性系统求解的高计算成本。尽管基于共轭梯度（CG）的无矩阵NGD方法避免了显式矩阵求逆，但格拉姆矩阵的病态性会显著减缓CG的收敛速度。本文将无矩阵NGD推广到更广泛的问题类别，并提出使用随机数值线性代数（RandNLA）技术对内层CG求解器进行高效预处理。所提出的算法在多种基于神经网络离散的偏微分方程问题上，相比现有NGD方法及其他先进优化器展现出显著性能提升。
原文摘要 · Abstract (English)
Natural Gradient Descent (NGD) has emerged as a promising optimization algorithm for training neural network-based solvers for partial differential equations (PDEs), such as Physics-Informed Neural Networks (PINNs). However, its practical use is often limited by the high computational cost of solving linear systems involving the Gramian matrix. While matrix-free NGD methods based on the conjugate gradient (CG) method avoid explicit matrix inversion, the ill-conditioning of the Gramian significantly slows the convergence of CG. In this work, we extend matrix-free NGD to broader classes of problems than previously considered and propose the use of Randomized Numerical Linear Algebra (RandNLA) techniques for efficient preconditioning of the inner CG solver. The resulting algorithm demonstrates substantial performance improvements over existing NGD-based methods and other state-of-the-art optimizers on a range of PDE problems discretized using neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。