SAMPa让SAM优化器并行化，速度翻倍且效果更好。
SAMPa: Sharpness-aware Minimization Parallelized
- 将SAM的两次梯度计算改为完全并行，提升效率。
- 实测速度比原SAM快一倍，且在视觉语言任务上表现更优。
- 理论保证固定扰动下仍能收敛，适合追求高效可靠的优化研究者。
Sharpness-aware minimization (SAM) 被证明能提升神经网络的泛化能力。但每次SAM更新需顺序计算两次梯度，相比SGD等基础优化器，每轮代价翻倍。我们提出SAM的简单改进SAMPa，可完全并行化两次梯度计算。在设备间通信成本可忽略的前提下，SAMPa实现两倍于SAM的速度提升。实验表明，SAMPa在计算时间上位列SAM最高效的变体之一。此外，其在视觉和语言任务中均持续优于SAM。值得注意的是，通过新颖的李雅普诺夫函数，我们理论上证明了即使扰动大小固定，SAMPa仍保持收敛性。我们正是以这一收敛保证为硬约束推导出SAMPa，该思路对开发SAM类方法具有普遍启发意义。代码已开源：https://github.com/LIONS-EPFL/SAMPa。
原文摘要 · Abstract (English)
Sharpness-aware minimization (SAM) has been shown to improve the generalization of neural networks. However, each SAM update requires \emph{sequentially} computing two gradients, effectively doubling the per-iteration cost compared to base optimizers like SGD. We propose a simple modification of SAM, termed SAMPa, which allows us to fully parallelize the two gradient computations. SAMPa achieves a twofold speedup of SAM under the assumption that communication costs between devices are negligible. Empirical results show that SAMPa ranks among the most efficient variants of SAM in terms of computational time. Additionally, our method consistently outperforms SAM across both vision and language tasks. Notably, SAMPa theoretically maintains convergence guarantees even for \emph{fixed} perturbation sizes, which is established through a novel Lyapunov function. We in fact arrive at SAMPa by treating this convergence guarantee as a hard requirement -- an approach we believe is promising for developing SAM-based methods in general. Our code is available at \url{https://github.com/LIONS-EPFL/SAMPa}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。