提出梯度下降在过参数线性模型中的局部收敛理论,放宽了对步长、宽度和初始化的要求。
A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Models
- 基于权重相关的局部PL条件与下降引理,建立非凸问题的收敛分析框架。
- 证明梯度下降在一般损失下实现线性收敛,且收敛速率依赖于初始权重和当前损失。
- 为步长选择提供新依据,适用于深度学习中过参数化模型的优化分析。
以往关于过参数化神经网络梯度下降(GD)收敛性的研究多依赖于强假设:步长极小、隐层宽度无限或初始化较大、谱范数平衡等。近期工作尝试放松这些假设,主要集中在带平方损失的两层线性网络上。本文在更一般损失函数下,针对两层线性神经网络,以更宽松的步长、宽度和初始化假设,推导出梯度下降的线性收敛率。关键挑战在于,经典非凸优化中的Polyak-Łojasiewicz(PL)条件与下降引理无法在全局成立。本文证明这两个条件可在局部成立,且局部常数依赖于权重。进一步给出了这些局部常数的上界,其依赖于权重初始化、当前损失以及非过参数化模型的全局PL常数和光滑性常数。基于此,我们推导出梯度下降的线性收敛速率。该分析不仅优于先前结果,还建议了更优的步长选择,并通过数值实验验证。
原文摘要 · Abstract (English)
Most prior work on the convergence of gradient descent (GD) for overparameterized neural networks relies on strong assumptions on the step size (infinitesimal), the hidden-layer width (infinite), or the initialization (large, spectral, balanced). Recent efforts to relax these assumptions focus on two-layer linear networks trained with the squared loss. In this work, we derive a linear convergence rate for training two-layer linear neural networks with GD for general losses and under relaxed assumptions on the step size, width, and initialization. A key challenge in deriving this result is that classical ingredients for deriving convergence rates for nonconvex problems, such as the Polyak-Łojasiewicz (PL) condition and Descent Lemma, do not hold globally for overparameterized neural networks. Here, we prove that these two conditions hold locally with local constants that depend on the weights. Then, we provide bounds on these local constants, which depend on the initialization of the weights, the current loss, and the global PL and smoothness constants of the non-overparameterized model. Based on these bounds, we derive a linear convergence rate for GD. Our convergence analysis not only improves upon prior results but also suggests a better choice for the step size, as verified through our numerical experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。