改进自适应梯度法，让学习率更稳定。
AdaGrad-Diff: A New Version of the Adaptive Gradient Algorithm
- 用梯度差值的累积平方和调整学习率
- 在梯度变化小时不盲目减小步长
- 适合对学习率敏感的复杂训练场景
标准梯度方法对学习率选择极为敏感，通常需人工调参。自适应方法可缓解此问题，其中AdaGrad影响深远。本文提出一种类似AdaGrad的自适应算法，其更新机制基于连续梯度差值的累积平方范数，而非梯度范数本身。核心思想是：当梯度跨迭代变化较小时，学习率不会被过度缩减；而显著的梯度波动（反映曲率或不稳定性）则会自动降低学习率。数值实验表明，在多个实际应用场景中，该方法比AdaGrad更具鲁棒性。
原文摘要 · Abstract (English)
Vanilla gradient methods are often highly sensitive to the choice of stepsize, which typically requires manual tuning. Adaptive methods alleviate this issue and have therefore become widely used. Among them, AdaGrad has been particularly influential. In this paper, we propose an AdaGrad-style adaptive method in which the adaptation is driven by the cumulative squared norms of successive gradient differences rather than gradient norms themselves. The key idea is that when gradients vary little across iterations, the stepsize is not unnecessarily reduced, while significant gradient fluctuations, reflecting curvature or instability, lead to automatic stepsize damping. Numerical experiments demonstrate that the proposed method is more robust than AdaGrad in several practically relevant settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。