让优化器用负步长，提升神经网络训练效果
Don't Be So Positive: Negative Step Sizes in Second-Order Methods
- 引入负步长机制，利用负曲率信息改进优化方向
- 在广义条件下，负步长配合Wolfe线搜索可全局收敛
- 实验表明比传统海森修正方法更有效，适合深度学习优化
二阶方法的优势在于利用曲率信息，但提取成本高，且获得的负曲率信息常被丢弃以保证全局收敛性，这限制了其在现代机器学习中的应用。本文表明，只要引入一个关键成分——负步长，二阶及类似方法对神经网络优化极具潜力。在非常一般的条件下，生成上升方向的方法结合允许正负步长的Wolfe线搜索时，仍可实现全局收敛。实验表明，使用负步长通常比常见的海森修正方法更有效。
原文摘要 · Abstract (English)
The value of second-order methods lies in the use of curvature information. Yet, this information is costly to extract and once obtained, valuable negative curvature information is often discarded so that the method is globally convergent. This limits the effectiveness of second-order methods in modern machine learning. In this paper, we show that second-order and second-order-like methods are promising optimizers for neural networks provided that we add one ingredient: negative step sizes. We show that under very general conditions, methods that produce ascent directions are globally convergent when combined with a Wolfe line search that allows both positive and negative step sizes. We experimentally demonstrate that using negative step sizes is often more effective than common Hessian modification methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。