SAM通过平衡性正则化提升模型鲁棒性，尤其适合含异常值的微调任务。
Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems
- 引入'平衡性'概念替代传统平滑度，刻画双变量系统的全局行为
- 理论与实证表明SAM促进平衡性，且对异常值响应更强
- 提出BAR方法，计算开销降低95%，性能优于SAM和SGD
Sharpness-aware minimization (SAM) 能够提升多种深度学习任务的泛化能力。受LoRA等流行架构启发，本文研究了SAM在涉及两组变量的尺度不变问题中的隐式正则化作用。不同于常关注的平滑度，本文提出'平衡性'概念，定义为两个变量平方范数之差，从而更丰富地描述SAM的全局行为。理论与实验结果表明：i) SAM促进平衡性；ii) 平衡性正则化具有数据响应性——异常值影响更强。这一现象与实际观察一致：当存在异常值时，SAM优于SGD。基于此隐式正则化，本文设计了一种资源高效型SAM变体，即平衡性感知正则化（BAR），专用于尺度不变问题，如使用LoRA微调语言模型。BAR相较SAM节省95%计算开销，在RoBERTa、GPT2及OPT-1.3B多个任务上表现更优。
原文摘要 · Abstract (English)
Sharpness-aware minimization (SAM) improves generalization of various deep learning tasks. Motivated by popular architectures such as LoRA, we explore the implicit regularization of SAM for scale-invariant problems involving two groups of variables. Instead of focusing on commonly used sharpness, this work introduces a concept termed balancedness, defined as the difference between the squared norm of two variables. This allows us to depict richer global behaviors of SAM. In particular, our theoretical and empirical findings reveal that i) SAM promotes balancedness; and ii) the regularization on balancedness is data-responsive -- outliers have stronger impact. The latter coincides with empirical observations that SAM outperforms SGD in the presence of outliers. Leveraging the implicit regularization, we develop a resource-efficient SAM variant, balancedness-aware regularization (BAR), tailored for scale-invariant problems such as finetuning language models with LoRA. BAR saves 95% computational overhead of SAM, with enhanced test performance across various tasks on RoBERTa, GPT2, and OPT-1.3B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。