数据几何决定神经网络泛化能力,越难被激活阈值分割越能泛化。
Generalization Below the Edge of Stability: The Role of Data Geometry
- 基于低维球面混合分布的理论推导,证明泛化误差随内在维数自适应下降。
- 数据集中在单位球面时,泛化性能变差,与实验观察一致。
- 揭示梯度下降隐式偏好:数据越难‘破碎’,越易学共性模式,泛化越好。
理解过参数化神经网络的泛化能力,关键在于数据几何、神经架构与训练动态之间的相互作用。本文针对在稳定性边缘以下训练的过参数化两层ReLU网络,进行理论分析。首先,对于支持在低维球面混合上的数据分布,推导出可证明自适应于内在维度的泛化界。其次,针对一类各向同性分布(其概率质量集中程度不同),得到一系列边界,显示当概率质量更集中于单位球面时,泛化率恶化。这些结果体现一个统一原理:若数据难以被ReLU神经元的激活阈值‘破碎’,梯度下降倾向于学习共享模式,从而获得良好泛化解;反之,若数据易被破碎(如分布在球面上),则趋向于记忆。理论结果整合了文献中分散的实证发现。
原文摘要 · Abstract (English)
Understanding generalization in overparameterized neural networks hinges on the interplay between the data geometry, neural architecture, and training dynamics. In this paper, we theoretically explore how data geometry controls this implicit bias. This paper presents theoretical results for overparametrized two-layer ReLU networks trained below the edge of stability. First, for data distributions supported on a mixture of low-dimensional balls, we derive generalization bounds that provably adapt to the intrinsic dimension. Second, for a family of isotropic distributions that vary in how strongly probability mass concentrates toward the unit sphere, we derive a spectrum of bounds showing that rates deteriorate as the mass concentrates toward the sphere. These results instantiate a unifying principle: When the data is harder to "shatter" with respect to the activation thresholds of the ReLU neurons, gradient descent tends to learn representations that capture shared patterns and thus finds solutions that generalize well. On the other hand, for data that is easily shattered (e.g., data supported on the sphere) gradient descent favors memorization. Our theoretical results consolidate disparate empirical findings that have appeared in the literature.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。