发现即使损失函数是二次的，梯度下降也能在稳定边缘收敛。
Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regime
- 用参数化线性回归模型验证：二次损失下仍存在稳定边缘现象。
- 实证与理论结合，证明梯度下降非渐近收敛到线性插值解。
- 揭示深度2对角线网络在大步长下的隐式偏差，拓展实用模型理解。
经典优化理论要求小步长以保证梯度下降（GD）收敛，但近期研究发现，即使步长η超过2/L（L为全局光滑常数），GD仍可收敛，此即‘稳定边缘’（EoS）现象。传统观点认为，子二次增长的目标函数是诱发EoS的关键。本文通过研究带参数化β = w⁺² - w⁻²的线性插值回归问题，发现当损失函数l(·)为二次时，在特定条件下仍会出现EoS。这一结论通过实证和理论双重验证，表明在非渐近意义下，GD轨迹可收敛至线性插值解。此外，该模型对应深度为2的对角线线性网络，在稳定边缘下长期未被充分探索。本研究揭示了大步长下对角线网络的隐式偏差，丰富了对更实际模型中稳定边缘现象的理解。
原文摘要 · Abstract (English)
Classical optimization theory requires a small step-size for gradient-based methods to converge. Nevertheless, recent findings challenge the traditional idea by empirically demonstrating Gradient Descent (GD) converges even when the step-size $η$ exceeds the threshold of $2/L$, where $L$ is the global smooth constant. This is usually known as the Edge of Stability (EoS) phenomenon. A widely held belief suggests that an objective function with subquadratic growth plays an important role in incurring EoS. In this paper, we provide a more comprehensive answer by considering the task of finding linear interpolator $β\in R^{d}$ for regression with loss function $l(\cdot)$, where $β$ admits parameterization as $β= w^2_{+} - w^2_{-}$. Contrary to the previous work that suggests a subquadratic $l$ is necessary for EoS, our novel finding reveals that EoS occurs even when $l$ is quadratic under proper conditions. This argument is made rigorous by both empirical and theoretical evidence, demonstrating the GD trajectory converges to a linear interpolator in a non-asymptotic way. Moreover, the model under quadratic $l$, also known as a depth-$2$ diagonal linear network, remains largely unexplored under the EoS regime. Our analysis then sheds some new light on the implicit bias of diagonal linear networks when a larger step-size is employed, enriching the understanding of EoS on more practical models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。