研究梯度下降中优化曲线是否凸，发现步长选择决定曲线凸性。
Are Convex Optimization Curves Convex?
- 通过分析步长影响，揭示优化曲线凸性的关键条件。
- 在保证单调收敛的步长范围内，部分情况可保证曲线凸性。
- 结果适用于梯度流，也解释了梯度范数是否单调下降。
本文研究梯度下降产生的优化曲线何时为凸——这能避免初始平台期后突然下降等难判断停止时机的现象。尽管一般函数优化可能出现此类行为，但在平滑凸函数这一理想情形下是否也会发生？此前研究未涉及此问题。我们发现，答案关键取决于步长选择。具体而言，在保证单调收敛到最优值的步长范围内，我们刻画出优化曲线可被严格证明凸的情形，以及可能非凸的情形。我们还将结果拓展至梯度流，并探讨了梯度范数是否单调递减这一相关但不同的问题。
原文摘要 · Abstract (English)
In this paper, we study when we might expect the optimization curve induced by gradient descent to be \emph{convex} -- precluding, for example, an initial plateau followed by a sharp decrease, making it difficult to decide when optimization should stop. Although such undesirable behavior can certainly occur when optimizing general functions, might it also occur in the benign and well-studied case of smooth convex functions? As far as we know, this question has not been tackled in previous work. We show, perhaps surprisingly, that the answer crucially depends on the choice of the step size. In particular, for the range of step sizes which are known to result in monotonic convergence to an optimal value, we characterize a regime where the optimization curve will be provably convex, and a regime where the curve can be non-convex. We also extend our results to gradient flow, and to the closely-related but different question of whether the gradient norm decreases monotonically.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。