激活函数影响神经ODE训练，光滑性和非线性决定全局收敛。
Global Convergence in Neural ODEs: Impact of Activation Functions
- 用光滑性和非线性激活函数保证前向后向微分方程解唯一。
- 在过参数化下，梯度下降可实现全局收敛，与神经正切核谱特性相关。
- 为实际应用提供调参指导，提升训练速度与性能。
神经常微分方程（Neural ODEs）因其连续性与参数共享效率，在多个领域表现优异。然而，其独特结构也带来了训练挑战，尤其体现在梯度计算精度和收敛性分析方面。本文研究激活函数的影响，发现其平滑性与非线性对训练动态至关重要：平滑激活函数能确保前向与反向ODE的全局唯一解，而足够的非线性有助于维持训练过程中神经正切核（NTK）的谱特性。结合这两点，我们在过参数化设定下证明了梯度下降可实现神经ODE的全局收敛。理论结果通过数值实验验证，不仅支持分析结论，还为神经ODE的扩展提供了实用指导，有望加速训练并提升真实场景中的性能。
原文摘要 · Abstract (English)
Neural Ordinary Differential Equations (ODEs) have been successful in various applications due to their continuous nature and parameter-sharing efficiency. However, these unique characteristics also introduce challenges in training, particularly with respect to gradient computation accuracy and convergence analysis. In this paper, we address these challenges by investigating the impact of activation functions. We demonstrate that the properties of activation functions, specifically smoothness and nonlinearity, are critical to the training dynamics. Smooth activation functions guarantee globally unique solutions for both forward and backward ODEs, while sufficient nonlinearity is essential for maintaining the spectral properties of the Neural Tangent Kernel (NTK) during training. Together, these properties enable us to establish the global convergence of Neural ODEs under gradient descent in overparameterized regimes. Our theoretical findings are validated by numerical experiments, which not only support our analysis but also provide practical guidelines for scaling Neural ODEs, potentially leading to faster training and improved performance in real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。