针对非光滑优化中的梯度震荡问题,提出自适应稳定的新算法。
Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization

- 通过随机方向导数方差估算局部几何不稳定性,动态调整步长。
- 在小批量和量化训练中提升准确率,最高增益达4.54%。
- 适合处理带ReLU或量化操作的深度模型训练,改善收敛性。
深度学习优化依赖于平滑损失曲面的假设,但现代架构中的ReLU激活和量化操作等非光滑组件常导致该假设失效。在此类非光滑场景下,Adam等自适应优化器因克拉克次微分内信号冲突而产生梯度振荡,导致收敛差、泛化性能不佳。为此,我们提出S-Adam,一种基于局部几何不稳定性(LGI)度量的新型优化器。LGI通过随机方向导数方差高效估算克拉克次微分直径,并引入自适应阻尼项exp(-λρ),在高不稳定性区域减缓更新,同时保持在平滑区域的快速收敛。理论分析表明,S-Adam以最优O(1/√T)速率几乎必然收敛至(δ,ε)-克拉克平稳点。在量化感知训练(QAT)和高噪声小批量学习上的实验显示,S-Adam持续优于AdamW和Prox-SGD,CIFAR-100上最高提升4.54%,TinyImageNet上提升4.27%,有效抑制梯度振荡。
原文摘要 · Abstract (English)
Deep learning optimization relies heavily on the assumption of smooth loss landscapes, a condition systematically violated by modern architectures due to non-smooth components such as ReLU activations and quantization operators. In such non-smooth regimes, adaptive optimizers such as Adam suffer from gradient chattering, violent oscillations caused by conflicting signals within the Clarke subdifferential, leading to poor convergence and suboptimal generalization. To address this, we introduce Singularity-aware Adam (S-Adam), a novel optimizer that stabilizes training by dynamically modulating step sizes based on local geometric instability. Our key contribution is the Local Geometric Instability (LGI) metric, a computationally efficient estimator of the Clarke subdifferential diameter derived from the variance of randomized directional derivatives. S-Adam incorporates an adaptive damping mechanism exp(-$λ$$ρ$) that decelerates updates in high-instability regions while preserving fast convergence in smooth basins. We provide a rigorous convergence analysis using differential inclusions, proving that S-Adam converges almost surely to ($δ$,$ε$)-Clarke stationary points at the optimal O(1/$\sqrt(T)$) rate. Empirical evaluations on Quantization-Aware Training (QAT) and high-noise small-batch learning demonstrate that S-Adam consistently outperforms AdamW and Prox-SGD, achieving accuracy gains of up to +4.54% on CIFAR-100 and +4.27% on TinyImageNet while effectively mitigating gradient oscillations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。