Ada2MS融合动量与AdamW优势，自动平衡局部与全局梯度信息。
Ada2MS: A Hybrid Optimization Algorithm Based on Exponential Mixing of Elementwise and Global Second-Moment Estimates
- 通过指数混合局部与全局二阶矩，实现平滑过渡的优化策略。
- 在视觉任务中表现媲美主流优化器，且对超参数更鲁棒。
- 适合追求稳定训练与良好泛化的深度学习研究者使用。
优化算法是机器学习模型迭代最小化损失函数、更新参数、从数据中学习并提升性能的核心方法。动量SGD与AdamW代表两种重要优化范式：AdamW更新稳定，跨训练场景鲁棒性较强，但泛化性能常弱于动量方法；动量SGD经精细调优后通常获得更好泛化效果，但对梯度尺度变化和超参数设置更敏感。为调和两者优劣，本文提出Ada2MS，一种基于元素级与全局二阶矩估计间连续指数插值的混合优化算法，实现从AdamW式行为到动量SGD式行为的平滑过渡。在本研究评估的视觉任务中，Ada2MS在统一比较协议下取得具有竞争力的结果。代码将发布于 https://github.com/mengzhu0308/Ada2MS。
原文摘要 · Abstract (English)
Optimization algorithms are core methods by which machine learning models iteratively minimize loss functions, update parameters, learn from data, and improve performance. Momentum SGD and AdamW represent two important optimization paradigms. AdamW produces stable updates and usually has strong robustness across training scenarios, but its generalization performance is sometimes weaker than that of momentum methods. Momentum SGD can often obtain better generalization after careful tuning, but it is more sensitive to gradient-scale variation and hyperparameter settings. To balance the strengths and weaknesses of the two paradigms, this paper proposes Ada2MS, an optimization algorithm that achieves a smooth transition between AdamW-like behavior and momentum-SGD-like behavior through continuous exponential interpolation between elementwise second-moment estimates and global second-moment estimates. On the visual tasks evaluated in this study, Ada2MS obtains competitive results under a unified optimizer-comparison protocol. The code will be released at https://github.com/mengzhu0308/Ada2MS
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。