证明了任意深度网络在普通激活函数下梯度下降收敛，无需特殊初始化。
Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness
- 基于广义Lipschitz光滑性，建立参数空间与梯度变化的控制关系。
- 在学习率足够小时，损失函数随迭代次数以1/T^{1/L}速率下降。
- 适用于多层线性、tanh等常见激活函数，适合理论研究者阅读。
本文为任意宽度或深度的前馈神经网络建立了梯度下降的收敛性保证，无需对初始化或数据集施加特殊要求。仅假设激活函数满足Lipschitz光滑、Lipschitz连续且线性有界——该性质对线性、tanh、softplus和sigmoid均成立。对于损失函数，要求其在模型输出上Lipschitz光滑，这在均方误差下成立。核心理论洞察是：激活函数的Lipschitz性质在多次复合后仍部分保持，从而导出一种新型广义Lipschitz光滑性条件，即梯度变化被参数空间变化乘以参数范数的多项式项所控制。该条件同时适用于模型函数与损失函数，进而推导出下降引理：当学习率相对于参数范数足够小时，损失递减。通过控制参数范数不快速增长至无穷，证明了在T次迭代内最小梯度范数以速率O(1/T^{1/L})收敛至零，其中L为网络层数。
原文摘要 · Abstract (English)
We establish convergence guarantees of gradient descent for general feedforward neural networks of arbitrary width or depth, with no special requirements on the initialization or dataset. We only assume that the activation functions are Lipschitz smooth, Lipschitz continuous, and linearly bounded--- properties that hold for linear, tanh, softplus, and sigmoid activation functions. For the loss function, we require that it is Lipschitz smooth in the model outputs, which is true for mean-squared error. The key theoretical insight is that the Lipschitz properties of the activation functions are partially preserved even through repeated compositions, leading to a novel generalized Lipschitz smoothness condition where the change in gradient is upper bounded by the change in the parameter space, multiplied by polynomial terms of the parameter norms at both endpoints. This type of condition holds for both the model function and the loss function, enabling a descent lemma where the loss decreases as long as the learning rate is small enough with respect to the parameter norms. By ensuring that the parameter norms do not grow too quickly to infinity, we prove that the minimum squared gradient norm converges to zero in $T$ iterations at rate $O(1/T^{1/L})$ for an $L$-layer neural network.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。