用精确曲率信息训练神经网络会失败，揭示了损失景观的新特性。
Nonlinear discretizations and Newton's method: characterizing stationary points of regression objectives
- 采用真实海森矩阵替代近似曲率进行优化
- 精确曲率导致训练可靠失败，无法收敛
- 挑战了局部极小值普遍存在的传统认知
二阶优化方法正成为梯度下降和ADAM等一阶优化器的有力替代方案。尽管科学机器学习领域普遍认可在优化步骤中引入曲率信息的优势，但已有研究仅限于拟牛顿法，即对目标函数的海森矩阵进行近似。然而，我们发现使用真实海森矩阵反而会导致神经网络训练完全失败。这种失败模式揭示了非线性离散化结构的几何特性以及损失景观中驻点的分布规律，从而质疑了‘损失景观充满局部极小值’这一普遍观点。
原文摘要 · Abstract (English)
Second-order methods are emerging as promising alternatives to standard first-order optimizers such as gradient descent and ADAM for training neural networks. Though the advantages of including curvature information in computing optimization steps have been celebrated in the scientific machine learning literature, the only second-order methods that have been studied are quasi-Newton, meaning that the Hessian matrix of the objective function is approximated. Though one would expect only to gain from using the true Hessian in place of its approximation, we show that neural network training reliably fails when relying on exact curvature information. The failure modes provide insight both into the geometry of nonlinear discretizations as well as the distribution of stationary points in the loss landscape, leading us to question the conventional wisdom that the loss landscape is replete with local minima.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。