从频域视角解析动量法，提出动态调整的优化器FSGDM
On the Performance Analysis of Momentum Method: A Frequency Domain Perspective
- 将动量法视为可变滤波器，通过频域分析梯度成分
- 训练后期应抑制高频梯度，早期保留原梯度，渐进放大低频成分
- 设计自适应动量的FSGDM，在多个任务上优于传统动量方法
基于动量的优化器广泛用于神经网络训练，但动量系数的最优选择仍不明确，阻碍了对动量在随机梯度方法中作用的理解。本文提出一种频域分析框架，将动量法视为梯度的时变滤波器，调整动量系数即改变滤波特性。实验验证该视角，并揭示关键发现：训练后期高频梯度分量有害；初期保留原始梯度，训练中逐步放大低频梯度分量能提升性能。基于此，提出频率随机梯度下降动量（FSGDM），一种通过经验性动态幅度响应调节动量滤波特性的启发式优化器。实验表明，FSGDM显著优于传统动量优化器。
原文摘要 · Abstract (English)
Momentum-based optimizers are widely adopted for training neural networks. However, the optimal selection of momentum coefficients remains elusive. This uncertainty impedes a clear understanding of the role of momentum in stochastic gradient methods. In this paper, we present a frequency domain analysis framework that interprets the momentum method as a time-variant filter for gradients, where adjustments to momentum coefficients modify the filter characteristics. Our experiments support this perspective and provide a deeper understanding of the mechanism involved. Moreover, our analysis reveals the following significant findings: high-frequency gradient components are undesired in the late stages of training; preserving the original gradient in the early stages, and gradually amplifying low-frequency gradient components during training both enhance performance. Based on these insights, we propose Frequency Stochastic Gradient Descent with Momentum (FSGDM), a heuristic optimizer that dynamically adjusts the momentum filtering characteristic with an empirically effective dynamic magnitude response. Experimental results demonstrate the superiority of FSGDM over conventional momentum optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。