将动量视为残差驱动的修正项,提升优化方向与收敛性。
Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

- 基于残差惩罚分解,把动量看作乘子式修正
- 提出RADAR算法,在三类任务上超越主流优化器
- 适合研究优化器设计或需稳定训练的深度学习场景
基于动量的优化器在现代深度学习中广泛应用,但动量递归、更新几何与加速机制之间的关系仍不完全清晰。本文提出一种受ADMM启发的动量框架(AIM),通过残差惩罚变量分解,将动量解释为由分解残差驱动的乘子式修正。AIM从ADMM式的乘子更新中恢复梯度指数移动平均,并分离了实际优化器中常混杂的两个机制:残差惩罚决定更新几何,而目标相关子问题的近似决定加速形式。基于AIM,我们提出相对论自适应梯度下降与加速残差(RADAR)算法,结合相对论自适应几何、解耦残差修正与二阶动量滤波,改善更新方向与动量估计。通过带方差扰动的李雅普诺夫漂移分析建立了随机收敛性。在监督视觉学习、语言建模和强化学习任务上的实验表明,RADAR在多个强基准优化器上均实现一致提升。
原文摘要 · Abstract (English)
Momentum-based optimizers are widely used in modern deep learning, yet the relations among momentum recursion, update geometry, and acceleration remain only partially understood. We develop an $\textbf{A}$DMM-$\textbf{I}$nspired $\textbf{M}$omentum (AIM) framework based on residual-penalty variable splitting, which interprets momentum as a multiplier-like correction driven by the splitting residual. AIM recovers the exponential moving average of gradients from an ADMM-style multiplier update and separates two mechanisms that are usually intertwined in practical optimizers: the residual penalty determines the update geometry, whereas the approximation of the objective-related subproblem determines the acceleration form. Building on AIM, we propose $\textbf{R}$elativistic $\textbf{A}$daptive gradient $\textbf{D}$escent with $\textbf{A}$ccelerated $\textbf{R}$esidual (RADAR), which combines relativistic adaptive geometry, decoupled residual correction, and second-order momentum filtering to improve the update direction and momentum estimation. We establish stochastic convergence through a variance-perturbed Lyapunov drift analysis. Experiments on supervised vision learning, language modeling, and reinforcement learning show that RADAR achieves consistent improvements over strong adaptive optimizer baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。