LSAM让SAM在分布式训练中更快更准，不卡顿还更好用。
LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimization
- 把SAM的对抗性搜索与异步采样结合，避免同步等待
- 大批次训练下收敛速度更快，最终准确率更高
- 适合大规模分布式深度学习，尤其加速模型训练
尽管锐度感知最小化（SAM）通过同时优化损失和锐度提升了深度神经网络的泛化能力，但在分布式大批次训练中效率低下。本文提出景观平滑的SAM（LSAM），一种新型优化器，在保持SAM泛化优势的同时显著提升效率。LSAM将SAM的对抗性步骤与异步分布式采样策略融合，构建出一种异步分布式采样方案，生成平滑的锐度感知损失景观用于优化。该设计消除了同步瓶颈，加速了大批次收敛，并在最终准确率上优于数据并行的SAM。
原文摘要 · Abstract (English)
While Sharpness-Aware Minimization (SAM) improves generalization in deep neural networks by minimizing both loss and sharpness, it suffers from inefficiency in distributed large-batch training. We present Landscape-Smoothed SAM (LSAM), a novel optimizer that preserves SAM's generalization advantages while offering superior efficiency. LSAM integrates SAM's adversarial steps with an asynchronous distributed sampling strategy, generating an asynchronous distributed sampling scheme, producing a smoothed sharpness-aware loss landscape for optimization. This design eliminates synchronization bottlenecks, accelerates large-batch convergence, and delivers higher final accuracy compared to data-parallel SAM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。