arXiv:2604.06652cs.LGmath.OC2026-04中稿 · IJCNN 2026被引 1

FlowAdam通过动态融合梯度流与Adam,提升耦合参数优化效果。

FlowAdam: Implicit Regularization via Geometry-Aware Soft Momentum Injection

  • 引入基于微分方程的连续梯度流,动态调整优化路径。
  • 在低秩矩阵/张量恢复上误差降低10%-22%,协同过滤任务降6%。
  • 适合参数耦合强的任务,如张量分解、图神经网络等场景。

自适应动量方法如Adam采用基于指数移动平均的对角预条件器,其坐标系依赖性在参数耦合密集或旋转时表现不佳,如矩阵分解、张量分解和图神经网络。本文提出FlowAdam,一种融合普通微分方程(ODE)连续梯度流的混合优化器。当EMA统计量检测到优化困难时,自动切换至截断的ODE积分。核心贡献为软动量注入机制,在模式转换中融合ODE速度与Adam动量,防止训练崩溃。在耦合优化基准上,该方法提供隐式正则化,使低秩矩阵/张量恢复的测试误差降低10%-22%,Jester任务降低6%,优于调优后的Lion和AdaBelief,且在良好条件任务(如CIFAR-10)上性能与Adam相当。MovieLens-100K实验表明收益源于参数耦合而非偏差估计。消融研究显示软注入至关重要,硬替换将准确率从100%降至82.5%。

原文摘要 · Abstract (English)

Adaptive moment methods such as Adam use a diagonal, coordinate-wise preconditioner based on exponential moving averages of squared gradients. This diagonal scaling is coordinate-system dependent and can struggle with dense or rotated parameter couplings, including those in matrix factorization, tensor decomposition, and graph neural networks, because it treats each parameter independently. We introduce FlowAdam, a hybrid optimizer that augments Adam with continuous gradient-flow integration via an ordinary differential equation (ODE). When EMA-based statistics detect landscape difficulty, FlowAdam switches to clipped ODE integration. Our central contribution is Soft Momentum Injection, which blends ODE velocity with Adam's momentum during mode transitions. This prevents the training collapse observed with naive hybrid approaches. Across coupled optimization benchmarks, the ODE integration provides implicit regularization, reducing held-out error by 10-22% on low-rank matrix/tensor recovery and 6% on Jester (real-world collaborative filtering), also surpassing tuned Lion and AdaBelief, while matching Adam on well-conditioned workloads (CIFAR-10). MovieLens-100K confirms benefits arise specifically from coupled parameter interactions rather than bias estimation. Ablation studies show that soft injection is essential, as hard replacement reduces accuracy from 100% to 82.5%.

优化器梯度流隐式正则化耦合参数

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。