基于梯度更新对齐原则，自动选择最优动量超参数。
Greedy Alignment Principle for Optimizer Selection
- 将优化器视为因果滤波器，通过梯度自相关最大化损失下降率。
- 理论证明贪心最优解存在且对统计估计扰动有稳定性保证。
- 在图像分类等任务中动态动量策略优于固定超参数搜索结果。
近期研究表明，梯度更新对齐是调节优化器更新的有力信号，常可加速训练。本文将这一逐更新的启发式方法提升为数学严谨的优化器选择原则。通过将梯度与更新视为信号，优化器视为映射两者的因果滤波器，我们将优化器选择问题建模为在预设优化器族中最大化期望损失下降率。我们证明该目标恰好等于优化器滤波器与梯度自相关之间的内积，并证明了贪心最优解的存在性及其在梯度统计估计扰动下的稳定性界。针对基于动量的优化器，该理论推导出适用于SGD+Momentum和Adam/AdamW的简单动态动量选择规则。在图像分类、语言模型微调及视觉变换器微调任务上的实验表明，所提出的动态动量规则在性能上匹配或超越通过人工遍历找到的最佳固定超参数，显著减少了对冗长动量搜索的需求。代码已开源：https://github.com/ironjr/gap
原文摘要 · Abstract (English)
Recent works have shown that gradient-update alignment is a powerful signal for modulating optimizer updates, often leading to faster training. We promote this update-wise heuristic as a mathematically grounded principle for selecting and tuning optimizer hyperparameters. By treating gradients and updates as signals and an optimizer as a causal filter that maps between them, we formulate optimizer selection as maximizing the expected drop rate in loss over a prescribed family of optimizers. We show that this objective is exactly the inner product between the optimizer filter and the gradient autocorrelation, and prove that a greedy optimum exists and has a stability bound under perturbations of the estimated gradient statistics. Specializing in momentum-based optimizers, the theory yields simple dynamic momentum selection rules for both SGD+Momentum and Adam/AdamW. Experiments across image classification, language model fine-tuning, and vision transformer fine-tuning show that the resulting dynamic momentum rules match or improve upon the best fixed hyperparameters found via manual sweeps, reducing the need for exhaustive momentum sweeps. Code is available at https://github.com/ironjr/gap
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。