提出新方法让物理神经网络训练更快更省资源。
Dual Natural Gradient Descent for Scalable Training of Physics-Informed Neural Networks
- 在残差空间计算自然梯度，降低复杂度
- 百万参数级模型单卡训练，误差降1-3个数量级
- 适合大规模物理模拟的高效神经网络训练
自然梯度方法能显著加速物理信息神经网络（PINNs）的训练，但其高斯-牛顿更新需在参数空间求解，时间复杂度高达 $O(n^3)$，其中 $n$ 为网络可训练权重数。本文发现该更新可在更小的残差空间中实现，残差空间大小为 $m = igsum_γ N_γ d_γ$，其中每个残差类 $γ$（如PDE内部、边界、初始数据）贡献 $N_γ$ 个采样点，输出维度为 $d_γ$。基于此，提出双自然梯度下降（D-NGD）：在残差空间中计算高斯-牛顿步，以极低开销加入测地线加速修正，并提供针对中小规模 $m$ 的稠密直接求解器与针对大规模 $m$ 的奈斯特朗预处理共轭梯度求解器。实验表明，D-NGD 可将二阶 PINN 优化扩展至含 1280 万参数的网络，在单张 GPU 上实现自然梯度训练，最终 $L^2$ 误差比一阶方法（Adam、SGD）和拟牛顿方法低一到三个数量级。
原文摘要 · Abstract (English)
Natural-gradient methods markedly accelerate the training of Physics-Informed Neural Networks (PINNs), yet their Gauss--Newton update must be solved in the parameter space, incurring a prohibitive $O(n^3)$ time complexity, where $n$ is the number of network trainable weights. We show that exactly the same step can instead be formulated in a generally smaller residual space of size $m = \sum_γ N_γ d_γ$, where each residual class $γ$ (e.g. PDE interior, boundary, initial data) contributes $N_γ$ collocation points of output dimension $d_γ$. Building on this insight, we introduce \textit{Dual Natural Gradient Descent} (D-NGD). D-NGD computes the Gauss--Newton step in residual space, augments it with a geodesic-acceleration correction at negligible extra cost, and provides both a dense direct solver for modest $m$ and a Nystrom-preconditioned conjugate-gradient solver for larger $m$. Experimentally, D-NGD scales second-order PINN optimization to networks with up to 12.8 million parameters, delivers one- to three-order-of-magnitude lower final error $L^2$ than first-order methods (Adam, SGD) and quasi-Newton methods, and -- crucially -- enables natural-gradient training of PINNs at this scale on a single GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。