动态调整梯度更新中的动量，提升训练稳定性和效果。
Dynamic Momentum Recalibration in Online Gradient Learning
- 用信号滤波思想在线调节动量增益，自适应优化梯度估计。
- 在多个模型和数据集上表现优于传统动量方法，媲美甚至超越顶尖优化器。
- 适用于多种优化框架，适合追求训练稳定性的深度学习研究者。
随机梯度下降（SGD）及其动量变体是深度学习优化的核心，但其梯度行为的内在机制仍不清晰。本文从信号处理视角重新审视梯度更新，发现固定动量系数会破坏偏差与方差的平衡，导致参数更新偏离最优。为此，我们提出SGDF（带滤波的SGD），受最优线性滤波启发，通过最小化均方误差在线计算时变增益，动态优化梯度估计，实现噪声抑制与信号保留的最佳权衡。该方法可扩展至其他优化器，具有广泛适用性。大量实验表明，SGDF在多种架构和基准上均优于传统动量方法，性能达到或超过现有最优优化器。
原文摘要 · Abstract (English)
Stochastic Gradient Descent (SGD) and its momentum variants form the backbone of deep learning optimization, yet the underlying dynamics of their gradient behavior remain insufficiently understood. In this work, we reinterpret gradient updates through the lens of signal processing and reveal that fixed momentum coefficients inherently distort the balance between bias and variance, leading to skewed or suboptimal parameter updates. To address this, we propose SGDF (SGD with Filter), an optimizer inspired by the principles of Optimal Linear Filtering. SGDF computes an online, time-varying gain to dynamically refine gradient estimation by minimizing the mean-squared error, thereby achieving an optimal trade-off between noise suppression and signal preservation. Furthermore, our approach could extend to other optimizers, showcasing its broad applicability to optimization frameworks. Extensive experiments across diverse architectures and benchmarks demonstrate SGDF surpasses conventional momentum methods and achieves performance on par with or surpassing state-of-the-art optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。