ReLU在物理信息学习中表现差，因自动微分无法处理不连续场的二阶导。
On the failure of ReLU activation for physics-informed machine learning
- 发现ReLU在仅含一阶导数的变分问题中仍失败
- 根源是PyTorch自动微分无法正确计算不连续场的二阶导
- 适用于研究物理信息神经网络训练机制的读者
物理信息机器学习利用常微分方程和/或偏微分方程训练神经网络以表示解场。与任何机器学习问题一样，激活函数的选择会影响解的特性与性能。多项研究对比了基准微分方程上的常见激活函数，一致发现修正线性单元（ReLU）劣于sigmoid、双曲正切和swish等函数。本文诊断了ReLU在物理信息学习中表现不佳的原因。尽管已知其分段线性形式导致其不适用于二阶微分方程，但我们发现，即使在仅涉及一阶导数的变分问题中，ReLU依然失败。我们识别出失败原因为激活函数的二阶导数——这些导数虽未出现在损失函数公式中，却在训练过程中通过自动微分引入。具体而言，PyTorch的自动微分无法准确刻画不连续场的导数，导致物理信息损失的梯度被错误指定，从而解释了ReLU的差表现。
原文摘要 · Abstract (English)
Physics-informed machine learning uses governing ordinary and/or partial differential equations to train neural networks to represent the solution field. Like any machine learning problem, the choice of activation function influences the characteristics and performance of the solution obtained from physics-informed training. Several studies have compared common activation functions on benchmark differential equations, and have unanimously found that the rectified linear unit (ReLU) is outperformed by competitors such as the sigmoid, hyperbolic tangent, and swish activation functions. In this work, we diagnose the poor performance of ReLU on physics-informed machine learning problems. While it is well-known that the piecewise linear form of ReLU prevents it from being used on second-order differential equations, we show that ReLU fails even on variational problems involving only first derivatives. We identify the cause of this failure as second derivatives of the activation, which are taken not in the formulation of the loss, but in the process of training. Namely, we show that automatic differentiation in PyTorch fails to characterize derivatives of discontinuous fields, which causes the gradient of the physics-informed loss to be mis-specified, thus explaining the poor performance of ReLU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。