arXiv:2606.28662cs.LGcs.AI2026-06

提出可解析计算的下降方向,让神经网络收敛到平坦最小值。

Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks

论文配图:Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks
图 1 · 摘自论文原文
  • 通过推导损失海森矩阵上界梯度,获得指向平坦最小值的解析方向。
  • 实验显示该方法能压缩海森特征值谱宽,避免尖锐极小点。
  • 首次实现无数值近似、闭式求解的平坦最小值优化方向,适合理论研究者。

平坦性假设认为,损失函数景观的平坦程度(由损失海森矩阵的特征值衡量)与神经网络泛化能力相关。尽管已有多种算法降低这些特征值,但多数集中于过程设计,未明确数据分布和网络参数如何结构性决定通向平坦最小值的方向。通常,此类方向的解析表征是难以处理的。为克服这一数学困难,近期研究推导出三层神经网络中交叉熵损失海森矩阵最大特征值的 Wolkowicz-Styan (WS) 上界。虽然该上界可微,但其梯度尚未被推导。因此,本文首次解析推导了该上界梯度,以刻画通向平坦最小值的方向。基于此,提出海森谱范围(HSR)正则化,沿WS上界的最陡下降方向更新参数。实验表明,该方法能缩小海森特征值谱范围,避开尖锐极小点与鞍点,促进收敛至平坦最小值。尽管当前仅适用于交叉熵损失和三层数架构,据作者所知,这是首项报告可闭式计算、无需数值近似的平坦最小值引导梯度的研究。该梯度的理论分析有望推动神经网络进一步发展。

原文摘要 · Abstract (English)

The flatness hypothesis suggests that flatness of the loss landscape, as measured by the eigenvalues of the loss Hessian, correlates with better neural network generalization. While various algorithms reduce these eigenvalues, most focus on procedural design, leaving it unclear how data distributions and NN parameters structurally determine directions toward flat minima. Characterizing these directions analytically is generally intractable. To overcome this mathematical difficulty, recent studies derived the Wolkowicz-Styan (WS) upper bound on the maximum eigenvalue of the cross-entropy loss Hessian in three-layer NNs. Although this upper bound is differentiable, its gradient was not derived. Therefore, we analytically derive the gradient of the WS upper bound to characterize directions leading to flat minima. Based on this, we propose Hessian Spectral Range (HSR) Regularization, which updates parameters along the steepest descent direction of the WS bound. Experiments demonstrate that HSR Regularization narrows the Hessian eigenvalue spectrum, avoids sharp minima and saddle points, and promotes convergence to flat minima. Although the applicability of this method is currently limited to cross-entropy loss and three-layer architectures, to the best of the authors' knowledge, this is the first study to report a closed-form gradient that promotes convergence to flat minima without numerical approximations. Therefore, the theoretical analysis of this gradient is expected to contribute to the further development of NNs.

神经网络平坦最小值海森矩阵正则化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。