提出轻量级分层梯度惩罚法，让SAM训练更快更省力。
Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learning
- 只对少数关键层惩罚梯度范数，大幅降低计算开销。
- 实验显示在图像与语言任务上性能媲美传统SAM。
- 理论证明局部扰动不损害收敛性，适合实际部署。
尖锐感知最小化（SAM）能提升神经网络泛化能力，但因模型扰动成本高，难以在真实场景中应用。现有改进方法虽试图降低开销，但效果有限。本文提出一种轻量级的分层梯度范数惩罚方法，在保持SAM优异泛化性能的同时显著降低计算代价。实验证明，仅对少数关键层进行梯度范数惩罚即可有效抑制全模型梯度范数。理论分析表明，这种局部扰动不会影响SAM的收敛速率，可安全应用于实际系统。我们在典型计算机视觉和语言建模基准上进行了广泛实验，结果表明该方法在性能上优于小批量SGD，且接近传统SAM水平。
原文摘要 · Abstract (English)
Sharpness-aware minimization (SAM) is known to improve the generalization performance of neural networks. However, it is not widely used in real-world applications yet due to its expensive model perturbation cost. A few variants of SAM have been proposed to tackle such an issue, but they commonly do not alleviate the cost noticeably. In this paper, we propose a lightweight layer-wise gradient norm penalizing method that tackles the expensive computational cost of SAM while maintaining its superior generalization performance. Our study empirically proves that the gradient norm of the whole model can be effectively suppressed by penalizing the gradient norm of only a few critical layers. We also theoretically show that such a partial model perturbation does not harm the convergence rate of SAM, allowing them to be safely adapted in real-world applications. To demonstrate the efficacy of the proposed method, we perform extensive experiments comparing the proposed method to mini-batch SGD and the conventional SAM using representative computer vision and language modeling benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。