提出快速计算神经网络高阶导数的新算法，显著提升求解微分方程效率。
A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks
- 基于TangentProp思想扩展出可计算任意阶导数的n-TangentProp方法
- 高阶导数计算时间从指数级降至准线性，支持深层宽网络
- 特别适合物理信息神经网络，训练加速明显且规模扩展更优
神经网络求解微分方程时，高阶导数的自动微分会导致运行时间呈指数增长。本文提出n-TangentProp，是TangentProp形式的自然推广，可对具有光滑无参数激活函数的全连接前馈神经网络f，在准线性时间内精确计算任意阶导数dⁿ/dxⁿ f(x)。我们在多种深度、宽度和导数阶数下进行了实证验证。结果表明，该方法在物理信息神经网络中尤其有效，相比以往方法显著缩短训练时间，并在模型规模和损失函数复杂度（由所需导数阶数决定）上表现出更优的可扩展性。代码已开源：https://github.com/kyrochi/n_tangentprop。
原文摘要 · Abstract (English)
The use of neural networks for solving differential equations is practically difficult due to the exponentially increasing runtime of autodifferentiation when computing high-order derivatives. We propose $n$-TangentProp, the natural extension of the TangentProp formalism \cite{simard1991tangent} to arbitrarily many derivatives. $n$-TangentProp computes the exact derivative $d^n/dx^n f(x)$ in quasilinear, instead of exponential time, for a densely connected, feed-forward neural network $f$ with a smooth, parameter-free activation function. We validate our algorithm empirically across a range of depths, widths, and number of derivatives. We demonstrate that our method is particularly beneficial in the context of physics-informed neural networks where \ntp allows for significantly faster training times than previous methods and has favorable scaling with respect to both model size and loss-function complexity as measured by the number of required derivatives. The code for this paper can be found at https://github.com/kyrochi/n\_tangentprop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。