用微分方程建模自适应优化算法，揭示其内在动态机制。
Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations
- 将AdaGrad、RMSProp、Adam统一为一阶积分微分方程。
- 数值模拟显示连续模型与离散算法行为高度一致。
- 适合研究优化理论或想深入理解自适应方法的读者。
本文提出一种连续时间框架，将AdaGrad、RMSProp和Adam优化算法建模为一阶积分微分方程。通过数值仿真及稳定性与收敛性分析，验证了这些方程作为原始算法的精确近似有效性。结果表明，连续模型的行为与离散实现之间具有高度一致性，为自适应优化方法提供了新的理论视角。
原文摘要 · Abstract (English)
In this paper, we propose a continuous-time formulation for the AdaGrad, RMSProp, and Adam optimization algorithms by modeling them as first-order integro-differential equations. We perform numerical simulations of these equations, along with stability and convergence analyses, to demonstrate their validity as accurate approximations of the original algorithms. Our results indicate a strong agreement between the behavior of the continuous-time models and the discrete implementations, thus providing a new perspective on the theoretical understanding of adaptive optimization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。