不同优化器在收敛时会自动筛选特征值，影响模型泛化能力。
Convergence of optimizers implies eigenvalues filtering at equilibrium
- 优化器通过超参数实现特征值过滤，决定寻找哪种极小值
- 新算法增强特征值筛选，更倾向于宽谷极小值（泛化更好）
- 理论适用于一般光滑函数，无需额外强假设，适合研究泛化机制
大量实验证据表明，深度神经网络训练中多种优化器趋向于接近全局最优。本文反向思考：假设优化器已收敛至任意点，分析其后果。结合近期关于‘稳定边界’现象的研究，我们提出不同优化器本质上是受超参数调控的特征值滤波器。标准梯度下降天然避开最尖锐的极小值，而尖锐感知最小化（SAM）算法进一步主动偏好更宽的盆地。基于此，我们提出两种新算法，具有更强的特征值过滤能力，有效促进更宽的极小值。理论分析基于广义Hadamard–Perron稳定流形定理，适用于一般的半代数C²函数，无需额外非退化条件或全局Lipschitz有界假设。我们在前馈神经网络上进行了数值实验验证结论。
原文摘要 · Abstract (English)
Ample empirical evidence in deep neural network training suggests that a variety of optimizers tend to find nearly global optima. In this article, we adopt the reversed perspective that convergence to an arbitrary point is assumed rather than proven, focusing on the consequences of this assumption. From this viewpoint, in line with recent advances on the edge-of-stability phenomenon, we argue that different optimizers effectively act as eigenvalue filters determined by their hyperparameters. Specifically, the standard gradient descent method inherently avoids the sharpest minima, whereas Sharpness-Aware Minimization (SAM) algorithms go even further by actively favoring wider basins. Inspired by these insights, we propose two novel algorithms that exhibit enhanced eigenvalue filtering, effectively promoting wider minima. Our theoretical analysis leverages a generalized Hadamard--Perron stable manifold theorem and applies to general semialgebraic $C^2$ functions, without requiring additional non-degeneracy conditions or global Lipschitz bound assumptions. We support our conclusions with numerical experiments on feed-forward neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。