提出平滑版符号优化器，解决参数差异导致的收敛问题。
Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
- 用温度控制的软符号变换替代硬符号更新，实现参数自适应调整。
- 在大模型预训练中显著优于传统符号优化器和AdamW，收敛更稳定。
- 适合需要低内存、高稳定性的大规模深度学习场景。
基于符号和LMO启发的优化器因性能强、内存占用低而受到广泛关注。然而，其固定幅度的更新会损害最终收敛性：更新机制与梯度大小脱钩，无法处理参数异质性，常导致振荡而非收敛。本文提出SoftSignum，通过温度控制的软符号变换替代硬符号映射，使参数级更新在符号型与梯度下降型之间平滑过渡。结合自适应分位数温度调度，并将该思想扩展至矩阵优化器，得到SoftMuon。此外，建立基于强凸正则化与Fenchel共轭的广义几何松弛框架，证明了在随机非凸设置下的收敛性。在多种深度学习任务（包括LLM预训练）上的实验表明，SoftSignum与SoftMuon持续优于对应的硬符号优化器及标准AdamW。
原文摘要 · Abstract (English)
Sign-based and LMO-inspired optimizers have recently attracted substantial attention in deep learning due to their strong performance and low memory footprint. However, their fixed-magnitude updates can hurt terminal convergence: they decouple update mechanisms from gradient magnitudes and fail to account for parameter heterogeneity, often leading to oscillation rather than convergence. We propose SoftSignum, a smooth relaxation of sign-based optimization that replaces the hard sign map with a temperature-controlled soft-sign transformation, enabling a parameter-wise transition from sign-like updates to magnitude-sensitive SGD-like steps. We complement it with an adaptive quantile-based temperature schedule and extend the same principle to matrix-valued optimizers, obtaining SoftMuon. We also develop a generalized geometry-relaxation framework based on strongly convex regularizers and Fenchel conjugates, proving convergence in stochastic non-convex setting. Experiments on diverse deep learning tasks, including LLM pretraining, show that SoftSignum and SoftMuon consistently improve over their hard sign-based counterparts and standard AdamW.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。