EAGLE通过分析梯度变化加速训练初期收敛，自动切换优化器提升稳定性。
eagle: early approximated gradient based learning rate estimator
- 利用连续步的参数与梯度变化估计损失曲面局部曲率，动态调整更新方向。
- 在标准数据集上，用更少轮次达到更快的损失下降速度，相比传统方法显著提速。
- 适合追求快速收敛的训练场景，尤其适用于训练初期阶段优化需求。
我们提出EAGLE更新规则，一种新颖的优化方法，通过利用当前及前一迭代步骤的参数和梯度值，在训练初期加速损失收敛。该算法通过计算连续训练步骤间参数与梯度的变化，并基于这些变化推导损失曲面的局部曲率来估计最优参数。然而，该更新规则存在潜在不稳定性，为此我们引入自适应切换机制，动态在Adam与EAGLE更新规则之间选择，以增强训练稳定性。在标准基准数据集上的实验表明，结合此新更新规则与切换机制的EAGLE优化器，相较于传统优化方法，能在更少的训练轮次内实现更快的损失收敛。
原文摘要 · Abstract (English)
We propose EAGLE update rule, a novel optimization method that accelerates loss convergence during the early stages of training by leveraging both current and previous step parameter and gradient values. The update algorithm estimates optimal parameters by computing the changes in parameters and gradients between consecutive training steps and leveraging the local curvature of the loss landscape derived from these changes. However, this update rule has potential instability, and to address that, we introduce an adaptive switching mechanism that dynamically selects between Adam and EAGLE update rules to enhance training stability. Experiments on standard benchmark datasets demonstrate that EAGLE optimizer, which combines this novel update rule with the switching mechanism achieves rapid training loss convergence with fewer epochs, compared to conventional optimization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。