arXiv:2506.11357cs.LGstat.ML2025-06NeurIPS被引 1

提出基于训练轨迹的泛化界,揭示梯度下降中损失梯度对泛化的影响。

Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel

论文配图:Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel
图 1 · 摘自论文原文
  • 引入损失路径核(LPK)捕捉全程训练轨迹,动态适应数据与优化过程。
  • 理论证明泛化误差受训练路径上损失梯度范数影响,且与核方法边界一致。
  • 适用于理解神经网络特征学习能力,适合关注泛化理论的研究者。

基于梯度的优化方法在实践中表现卓越,但其泛化性质仍缺乏完整理论解释。本文为梯度流建立了一个泛化界,该界与核方法的经典Rademacher复杂度边界一致,基于一种依赖数据的损失路径核(LPK)。与静态核(如NTK)不同,LPK能捕获整个训练轨迹,同时适应数据和优化动态,从而获得更紧致、更丰富的泛化保证。此外,该界揭示了训练路径上损失梯度范数对最终泛化性能的关键影响。证明核心结合了梯度流的稳定性分析与通过Rademacher复杂度的统一收敛。所提边界可恢复过参数化神经网络的已有核回归结果,并展现出神经网络相比核方法的特征学习能力。在真实数据集上的数值实验验证了该界与真实泛化差距具有良好的相关性。

原文摘要 · Abstract (English)

Gradient-based optimization methods have shown remarkable empirical success, yet their theoretical generalization properties remain only partially understood. In this paper, we establish a generalization bound for gradient flow that aligns with the classical Rademacher complexity bounds for kernel methods-specifically those based on the RKHS norm and kernel trace-through a data-dependent kernel called the loss path kernel (LPK). Unlike static kernels such as NTK, the LPK captures the entire training trajectory, adapting to both data and optimization dynamics, leading to tighter and more informative generalization guarantees. Moreover, the bound highlights how the norm of the training loss gradients along the optimization trajectory influences the final generalization performance. The key technical ingredients in our proof combine stability analysis of gradient flow with uniform convergence via Rademacher complexity. Our bound recovers existing kernel regression bounds for overparameterized neural networks and shows the feature learning capability of neural networks compared to kernel methods. Numerical experiments on real-world datasets validate that our bounds correlate well with the true generalization gap.

泛化界梯度流核方法神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。