提出可放宽学习率衰减的神经网络泛化理论，适用于常见深度模型。
Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networks
- 在齐次神经网络中证明可使用更慢的学习率衰减（Ω(1/√t)）
- 理论支持非凸优化下更优的学习率设置，不依赖严格递减
- 结果适用于含ReLU/LeakyReLU的全连接与卷积网络
算法稳定性是泛化分析中最有力的工具之一。然而，其推导通常要求在非凸训练环境下学习率按 η_t = O(1/t) 严格衰减，这种刚性设定可能影响优化性能且与实际不符。本文在齐次神经网络框架下推导泛化界，证明在温和假设下，学习率可采用更慢的衰减速度 Ω(1/√t)。我们进一步将理论拓展至非Lipschitz情形。该结论具有广泛适用性，因齐次神经网络涵盖使用ReLU和LeakyReLU激活函数的全连接与卷积神经网络。
原文摘要 · Abstract (English)
Algorithmic stability is among the most potent techniques in generalization analysis. However, its derivation usually requires a stepsize $η_t = \mathcal{O}(1/t)$ under non-convex training regimes, where $t$ denotes iterations. This rigid decay of the stepsize potentially impedes optimization and may not align with practical scenarios. In this paper, we derive the generalization bounds under the homogeneous neural network regimes, proving that this regime enables slower stepsize decay of order $Ω(1/\sqrt{t})$ under mild assumptions. We further extend the theoretical results from several aspects, e.g., non-Lipschitz regimes. This finding is broadly applicable, as homogeneous neural networks encompass fully-connected and convolutional neural networks with ReLU and LeakyReLU activations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。