改进SAM优化器，让模型训练更稳定高效
GCSAM: Gradient Centralized Sharpness Aware Minimization
- 用梯度归一化稳定训练过程，降低噪声影响
- 在图像分类与医学影像任务中通用性更强
- 比SAM和Adam更快收敛，且泛化性能更优
深度神经网络的泛化能力对模型在未见数据上的鲁棒性至关重要。近期研究表明，基于尖锐度的度量能通过引导模型收敛至平坦极小值来提升泛化性能。其中，尖锐度感知最小化（SAM）是一种有效的优化方法，可降低损失曲面的尖锐度，从而改善泛化。然而，SAM存在计算开销大、对噪声梯度敏感的问题，限制了其可扩展性和效率。为此，本文提出梯度集中式尖锐度感知最小化（GCSAM），引入梯度归一化（GC）以稳定梯度并加速收敛。GCSAM在上升步前对梯度进行归一化，减少噪声与方差，提升训练稳定性。实验表明，GCSAM在泛化性能和计算效率上均优于SAM和Adam，在通用图像与医学影像任务中表现一致优异。
原文摘要 · Abstract (English)
The generalization performance of deep neural networks (DNNs) is a critical factor in achieving robust model behavior on unseen data. Recent studies have highlighted the importance of sharpness-based measures in promoting generalization by encouraging convergence to flatter minima. Among these approaches, Sharpness-Aware Minimization (SAM) has emerged as an effective optimization technique for reducing the sharpness of the loss landscape, thereby improving generalization. However, SAM's computational overhead and sensitivity to noisy gradients limit its scalability and efficiency. To address these challenges, we propose Gradient-Centralized Sharpness-Aware Minimization (GCSAM), which incorporates Gradient Centralization (GC) to stabilize gradients and accelerate convergence. GCSAM normalizes gradients before the ascent step, reducing noise and variance, and improving stability during training. Our evaluations indicate that GCSAM consistently outperforms SAM and the Adam optimizer in terms of generalization and computational efficiency. These findings demonstrate GCSAM's effectiveness across diverse domains, including general and medical imaging tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。