提出Sassha优化器，让模型收敛到更平坦的解，提升泛化能力。
SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
- 通过显式降低解的尖锐度来改善泛化性能
- 在标准深度学习任务中表现优于或媲美现有方法
- 支持懒惰海塞矩阵更新，兼顾效率与稳定性
近似二阶优化方法常因泛化性能较差而逊于一阶方法。本文从损失曲面角度分析发现，现有二阶方法倾向于收敛到更尖锐的极小值。为此，我们提出Sassha，一种新型二阶优化算法，通过显式降低解的尖锐度来增强泛化能力，同时稳定优化轨迹中的近似海塞矩阵计算。该尖锐度最小化策略还兼容懒惰海塞更新机制，兼顾平坦性与计算效率。我们在大量标准深度学习实验中验证其有效性，Sassha展现出优异的泛化性能，优于或媲美其他方法。我们还进行了收敛性、鲁棒性、稳定性、效率和成本等方面的全面分析。
原文摘要 · Abstract (English)
Approximate second-order optimization methods often exhibit poorer generalization compared to first-order approaches. In this work, we look into this issue through the lens of the loss landscape and find that existing second-order methods tend to converge to sharper minima compared to SGD. In response, we propose Sassha, a novel second-order method designed to enhance generalization by explicitly reducing sharpness of the solution, while stabilizing the computation of approximate Hessians along the optimization trajectory. In fact, this sharpness minimization scheme is crafted also to accommodate lazy Hessian updates, so as to secure efficiency besides flatness. To validate its effectiveness, we conduct a wide range of standard deep learning experiments where Sassha demonstrates its outstanding generalization performance that is comparable to, and mostly better than, other methods. We provide a comprehensive set of analyses including convergence, robustness, stability, efficiency, and cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。