通过木伯里公式等技术，让物理信息神经网络训练快75倍
Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization
- 用木伯里公式降低能量自然梯度计算复杂度
- 结合动量与随机采样，使训练速度提升75倍
- 适合需要快速训练的物理建模场景
自然梯度方法能显著加速物理信息神经网络（PINNs）的训练，但计算成本过高。本文提出一系列改进能量自然梯度下降（ENGD）的方法：首先利用木伯里公式大幅降低计算复杂度；其次借鉴变分蒙特卡洛中的子采样投影增量自然梯度算法以加快收敛；最后探索在大批次下使用随机算法进一步降低计算开销。数值实验表明，所提方法在保持原版ENGD相同$L^2$误差的前提下，最快可实现75倍加速，且在低维问题早期训练中随机化效果显著；但在其他场景中存在关键瓶颈。
原文摘要 · Abstract (English)
Natural gradient methods significantly accelerate the training of Physics-Informed Neural Networks (PINNs), but are often prohibitively costly. We introduce a suite of techniques to improve the accuracy and efficiency of energy natural gradient descent (ENGD) for PINNs. First, we leverage the Woodbury formula to dramatically reduce the computational complexity of ENGD. Second, we adapt the Subsampled Projected-Increment Natural Gradient Descent algorithm from the variational Monte Carlo literature to accelerate the convergence. Third, we explore the use of randomized algorithms to further reduce the computational cost in the case of large batch sizes. We find that randomization accelerates progress in the early stages of training for low-dimensional problems, and we identify key barriers to attaining acceleration in other scenarios. Our numerical experiments demonstrate that our methods outperform previous approaches, achieving the same $L^2$ error as the original ENGD up to $75\times$ faster.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。