SODA统一了多种先进优化器，无需调参即可提升训练效果。
Optimistic Dual Averaging Unifies Modern Optimizers
- 提出SODA框架，统一Muons、Lion等优化器的理论视角。
- 引入1/k衰减策略，自动消除权重衰减调参需求。
- 在多尺度训练中表现更优，且无需额外调参，适合实操场景。
我们提出了SODA，即乐观对偶平均的泛化形式，为当前最先进的优化器（如Muons、Lion、AdEMAMix和NAdam）提供了统一视角，表明它们均可视为该框架的乐观实例。基于此框架，我们设计了一个适用于任意基础优化器的实用SODA封装，通过理论上合理的1/k衰减调度自动消除权重衰减调参需求。在多种规模和训练周期下的实验表明，SODA在不增加任何超参数调优的前提下，持续提升性能。
原文摘要 · Abstract (English)
We introduce SODA, a generalization of Optimistic Dual Averaging, which provides a common perspective on state-of-the-art optimizers like Muon, Lion, AdEMAMix and NAdam, showing that they can all be viewed as optimistic instances of this framework. Based on this framing, we propose a practical SODA wrapper for any base optimizer that eliminates weight decay tuning through a theoretically-grounded $1/k$ decay schedule. Empirical results across various scales and training horizons show that SODA consistently improves performance without any additional hyperparameter tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。