HGM优化器根据梯度方向一致性自适应调整学习率,提升训练稳定性与收敛速度。
Hindsight-Guided Momentum (HGM) Optimizer: An Approach to Adaptive Learning Rate
- 通过余弦相似度评估当前梯度与动量的对齐程度,动态调节学习率。
- 在平滑区域加速收敛,在震荡或噪声区域降低学习率保持稳定。
- 无需额外计算开销,适合深度神经网络等非凸优化场景。
我们提出一种一阶优化算法 Hindsight-Guided Momentum (HGM),基于近期更新的方向一致性自适应缩放学习率。传统自适应方法(如 Adam、RMSprop)仅依据梯度幅值调整学习动态,常忽略重要的几何线索。几何线索指梯度方向信息,例如当前梯度与历史更新之间的对齐程度,反映局部曲率和优化路径的一致性。HGM引入回溯机制,计算当前梯度与累积动量的余弦相似度,以区分一致与冲突的梯度方向:方向一致时提高学习率,震荡或噪声区域则降低学习率。该设计使优化器在损失曲面平滑区域加速收敛,同时在陡峭或不规则区域保持稳定性。尽管具备更强适应性,其计算与内存开销仍与现有优化器相当。通过更智能地响应优化空间结构,HGM在深度神经网络训练等非凸场景中提供了一种简单而有效的改进。
原文摘要 · Abstract (English)
We introduce Hindsight-Guided Momentum (HGM), a first-order optimization algorithm that adaptively scales learning rates based on the directional consistency of recent updates. Traditional adaptive methods, such as Adam or RMSprop , adapt learning dynamics using only the magnitude of gradients, often overlooking important geometric cues.Geometric cues refer to directional information, such as the alignment between current gradients and past updates, which reflects the local curvature and consistency of the optimization path. HGM addresses this by incorporating a hindsight mechanism that evaluates the cosine similarity between the current gradient and accumulated momentum. This allows it to distinguish between coherent and conflicting gradient directions, increasing the learning rate when updates align and reducing it in regions of oscillation or noise. The result is a more responsive optimizer that accelerates convergence in smooth regions of the loss surface while maintaining stability in sharper or more erratic areas. Despite this added adaptability, the method preserves the computational and memory efficiency of existing optimizers.By more intelligently responding to the structure of the optimization landscape, HGM provides a simple yet effective improvement over existing approaches, particularly in non-convex settings like that of deep neural network training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。