变量消除能化解非凸优化中的鞍点问题,提升收敛稳定性。
A Saddle Point Remedy: Power of Variable Elimination in Non-convex Optimization
- 通过消元重构优化景观,将鞍点转化为极小值点。
- 在矩阵分解和深层残差网络中显著改善收敛性与稳定性。
- 为设计更鲁棒的优化算法提供新思路,适合研究者参考。
鞍点而非劣质局部极小值,正成为大规模非凸优化中的主要障碍。尽管变量投影(VarPro)等变量消除算法在实践中表现出优异的收敛性和鲁棒性,但其原理一直未被充分理解。本文通过赫森矩阵惯性与舒尔补的严格分析,证明变量消除会根本性重塑目标函数的临界点结构:还原后的景观中的局部极大值,直接对应于原问题中的鞍点。这一发现通过典型非凸矩阵分解、双参数神经网络可视化及深层残差网络训练验证。结果表明,该方法在稳定性和收敛至更优极小值方面有显著提升。本工作不仅解释了现有方法,更确立了通过鞍点转换实现景观简化的普适原则,可指导新一代高效鲁棒优化算法的设计。
原文摘要 · Abstract (English)
The proliferation of saddle points, rather than poor local minima, is increasingly understood to be a primary obstacle in large-scale non-convex optimization for machine learning. Variable elimination algorithms, like Variable Projection (VarPro), have long been observed to exhibit superior convergence and robustness in practice, yet a principled understanding of why they so effectively navigate these complex energy landscapes has remained elusive. In this work, we provide a rigorous geometric explanation by comparing the optimization landscapes of the original and reduced formulations. Through a rigorous analysis based on Hessian inertia and the Schur complement, we prove that variable elimination fundamentally reshapes the critical point structure of the objective function, revealing that local maxima in the reduced landscape are created from, and correspond directly to, saddle points in the original formulation. Our findings are illustrated on the canonical problem of non-convex matrix factorization, visualized directly on two-parameter neural networks, and finally validated in training deep Residual Networks, where our approach yields dramatic improvements in stability and convergence to superior minima. This work goes beyond explaining an existing method; it establishes landscape simplification via saddle point transformation as a powerful principle that can guide the design of a new generation of more robust and efficient optimization algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。