用高阶微分方程求解器替代梯度下降，提升神经网络训练稳定性与效率
Learning by solving differential equations
- 将深度学习优化器升级为高阶龙格-库塔（RK）微分方程求解器
- 在多个数据集上实现比标准SGD更快收敛，且更稳定
- 适用于追求训练效率与鲁棒性的深度学习研究者
现代深度学习算法主要依赖梯度下降变体进行学习。梯度下降本质上是最简单的常微分方程（ODE）求解方法——即对梯度流方程应用欧拉法。自欧拉以来，人们设计出许多更精确、更稳定的高阶ODE求解器，如龙格-库塔（Runge-Kutta, RK）方法。然而，这些高阶方法尚未在深度学习中广泛应用。本文评估了将高阶RK求解器应用于深度学习的性能，分析其局限性，并提出克服方法。特别地，我们探索如何通过自然融入现代神经网络优化器的关键组件——如预条件、自适应学习率和动量——来提升其表现。
原文摘要 · Abstract (English)
Modern deep learning algorithms use variations of gradient descent as their main learning methods. Gradient descent can be understood as the simplest Ordinary Differential Equation (ODE) solver; namely, the Euler method applied to the gradient flow differential equation. Since Euler, many ODE solvers have been devised that follow the gradient flow equation more precisely and more stably. Runge-Kutta (RK) methods provide a family of very powerful explicit and implicit high-order ODE solvers. However, these higher-order solvers have not found wide application in deep learning so far. In this work, we evaluate the performance of higher-order RK solvers when applied in deep learning, study their limitations, and propose ways to overcome these drawbacks. In particular, we explore how to improve their performance by naturally incorporating key ingredients of modern neural network optimizers such as preconditioning, adaptive learning rates, and momentum.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。