提出一种无需梯度的平滑优化新方法，提升模型泛化能力。
Zeroth-Order Sharpness-Aware Learning with Exponential Tilting
- 用指数倾斜目标连接平均与最大损失，实现平滑过渡。
- 在分类、问答和生成任务中优于传统零阶基线。
- 适合无梯度或内存受限场景，兼具高效与鲁棒性。
经典零阶优化通常针对随机扰动参数下模型损失的期望值进行优化，即在扰动集合中使损失值平均较小。而主流的尖锐感知最小化（SAM）则关注邻域内最大损失，以更有效地找到平坦极小值。本文通过指数倾斜目标，显式连接零阶优化与SAM方法，提供从平均到最大损失的平滑过渡。我们提出了基于倾斜参数 $t$ 的软SAM零阶算法，并精确刻画了该框架下的尖锐性概念。实验表明，该方法可作为梯度无关且内存高效的SAM替代方案，在分类、多选问答和语言生成等任务上均优于普通零阶基线，实现更好泛化性能。
原文摘要 · Abstract (English)
Classic zeroth-order optimization approaches typically optimize for a smoothed version of the original function, i.e., the expected objective under randomly perturbed model parameters. This can be interpreted as encouraging the loss values in the perturbation set to be small on average. Popular sharpness-aware minimization (SAM) objectives, however, typically focus on the largest loss within the neighborhood to arrive at flat minima more effectively. In this work, we connect zeroth-order optimization (and its corresponding objectives) with SAM approaches explicitly, through an exponential tilting objective that provides a smooth transition between the average- and the max-loss formulations. We explore new zeroth-order algorithms to solve a soft SAM objective parameterized by a tilting parameter $t$. We provide precise characterizations of the sharpness notions of the tilted SAM framework. Practically, our approach can be used as a gradient-free and memory-efficient alternative to SAM variants, and it achieves better generalization compared to vanilla zeroth-order baselines on a wide range of downstream tasks, including classification, multiple choice QA, and language generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。