提出异步SAM，让模型训练更快更准，不增加时间开销。
Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning
- 异步并行处理，打破扰动与更新的数据依赖
- 在CIFAR-100上达到与原SAM相当的精度，训练时间接近SGD
- 可隐藏扰动耗时，充分调度CPU/GPU资源
Sharpness-Aware Minimization (SAM) 是一种提升模型泛化性能的优化方法。尽管其泛化能力优异，但因计算成本高，未被广泛应用于实际场景。本文提出一种新型异步并行SAM，可在几乎不增加计算代价的情况下，实现与原SAM相当的梯度范数惩罚效果，并打破模型扰动与更新之间的数据依赖关系。通过系统感知地调整扰动批次大小，该方法可完全隐藏模型扰动时间，从而充分调动异构系统资源（如CPU和GPU）。大量实验验证了其实际优势：例如，在CIFAR-100上，异步SAM的视觉变换器微调精度与原始SAM相当，而训练时间几乎与SGD相同。
原文摘要 · Abstract (English)
Sharpness-Aware Minimization (SAM) is an optimization method that improves generalization performance of machine learning models. Despite its superior generalization, SAM has not been actively used in real-world applications due to its expensive computational cost. In this work, we propose a novel asynchronous-parallel SAM which achieves nearly the same gradient norm penalizing effect like the original SAM while breaking the data dependency between the model perturbation and the model update. The proposed asynchronous SAM can even entirely hide the model perturbation time by adjusting the batch size for the model perturbation in a system-aware manner. Thus, the proposed method enables to fully utilize heterogeneous system resources such as CPUs and GPUs. Our extensive experiments well demonstrate the practical benefits of the proposed asynchronous approach. E.g., the asynchronous SAM achieves comparable Vision Transformer fine-tuning accuracy (CIFAR-100) as the original SAM while having almost the same training time as SGD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。