研究数据方向性如何影响线性网络训练与泛化性能
The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networks
- 用带尖峰协方差结构建模数据方向性，分析两层线性网络学习过程
- 发现训练分两阶段：先由输入输出相关性主导，后由数据主方向决定
- 给出泛化误差解析表达式，揭示数据结构与任务对齐提升性能
深度神经网络的成功很大程度上依赖于训练数据的统计结构。尽管在各向同性数据上的学习动态和泛化性能已有充分研究，但显著各向异性的影响尚未完全理解。本文在线性回归设置下，研究了由尖峰协方差结构（spiked covariance structure）表征的数据各向异性对两层线性网络学习动态与泛化误差的影响。分析表明，学习过程分为两个明显阶段：初始阶段受输入-输出相关性支配，后续阶段由数据结构的其他主方向主导。此外，我们推导出泛化误差的解析表达式，量化了数据尖峰结构与学习任务对齐程度对性能的提升作用。研究结果为理解数据各向异性如何塑造学习轨迹与最终表现提供了深刻的理论洞见，并为更复杂网络架构中的交互机制奠定了基础。
原文摘要 · Abstract (English)
The success of deep neural networks largely depends on the statistical structure of the training data. While learning dynamics and generalization on isotropic data are well-established, the impact of pronounced anisotropy on these crucial aspects is not yet fully understood. We examine the impact of data anisotropy, represented by a spiked covariance structure, a canonical yet tractable model, on the learning dynamics and generalization error of a two-layer linear network in a linear regression setting. Our analysis reveals that the learning dynamics proceed in two distinct phases, governed initially by the input-output correlation and subsequently by other principal directions of the data structure. Furthermore, we derive an analytical expression for the generalization error, quantifying how the alignment of the spike structure of the data with the learning task improves performance. Our findings offer deep theoretical insights into how data anisotropy shapes the learning trajectory and final performance, providing a foundation for understanding complex interactions in more advanced network architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。