提出新型扰动SGD，显著收紧非凸学习的泛化误差上界。
Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learning
- 引入类型II扰动SGD，分解泛化误差为轨迹项与平坦性项
- 轨迹项达O(n⁻¹)，优于此前O((nb)⁻⁰·⁵)的界限
- 适用于深度学习等场景，尤其关注模型泛化性能者必读
随机梯度下降（SGD）是训练深度神经网络的基础方法，尤其在非凸设置下至关重要。理解SGD的泛化能力对保证模型在未见数据上的鲁棒表现极为关键。本文通过引入类型II扰动SGD（T2pm-SGD），分析非凸学习中SGD的泛化误差边界，该方法可处理子高斯和有界损失函数。泛化误差被分解为轨迹项与平坦性项。我们的分析将轨迹项改进至O(n⁻¹)，显著优于此前针对有界损失的O((nb)⁻¹/²)结果，其中n为训练样本数，b为批量大小。通过选择最优扰动噪声方差，整体上界进一步优化至O(n⁻²/³)。对于子高斯损失函数，同样获得更紧的轨迹项。两种情况下，平坦性项在迭代过程中保持稳定，且小于以往文献中随迭代增长的结果。该稳定性由T2pm-SGD保障，从而为两类损失函数均带来更紧的泛化误差界。理论结果在MNIST与CIFAR-10等基准数据集上通过大量实验验证，表明T2pm-SGD在建立更紧泛化界方面的有效性。
原文摘要 · Abstract (English)
Stochastic Gradient Descent (SGD) is fundamental for training deep neural networks, especially in non-convex settings. Understanding SGD's generalization properties is crucial for ensuring robust model performance on unseen data. In this paper, we analyze the generalization error bounds of SGD for non-convex learning by introducing the Type II perturbed SGD (T2pm-SGD), which accommodates both sub-Gaussian and bounded loss functions. The generalization error bound is decomposed into two components: the trajectory term and the flatness term. Our analysis improves the trajectory term to $O(n^{-1})$, significantly enhancing the previous $O((nb)^{-1/2})$ bound for bounded losses, where n is the number of training samples and b is the batch size. By selecting an optimal variance for the perturbation noise, the overall bound is further refined to $O(n^{-2/3})$. For sub-Gaussian loss functions, a tighter trajectory term is also achieved. In both cases, the flatness term remains stable across iterations and is smaller than those reported in previous literature, which increase with iterations. This stability, ensured by T2pm-SGD, leads to tighter generalization error bounds for both loss function types. Our theoretical results are validated through extensive experiments on benchmark datasets, including MNIST and CIFAR-10, demonstrating the effectiveness of T2pm-SGD in establishing tighter generalization bounds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。