发现神经网络训练中存在锥形约束现象，提升模型性能。
On the Cone Effect in the Learning Dynamics
- 通过分析eNTK演化，揭示训练分两个阶段：初期剧烈变化，后期受锥形空间约束。
- 锥形约束阶段的模型性能显著优于完全线性化训练。
- 适合研究训练动态、优化机制的学者参考。
理解神经网络的学习动态是深度学习领域的核心问题。本文从实验视角研究真实场景下神经网络的学习动态，重点考察训练过程中经验神经正切核（eNTK）的演化过程。关键发现表明存在一个两阶段学习过程：第一阶段，eNTK发生显著变化，处于丰富区间；第二阶段，尽管eNTK仍在演化，但被限制在狭窄空间内，我们称之为锥形效应。这一两阶段框架建立在Fort等（2020）提出的假设基础上，但首次在实证中识别出第二阶段的锥形效应，并证明其相较于完全线性化训练具有显著性能优势。
原文摘要 · Abstract (English)
Understanding the learning dynamics of neural networks is a central topic in the deep learning community. In this paper, we take an empirical perspective to study the learning dynamics of neural networks in real-world settings. Specifically, we investigate the evolution process of the empirical Neural Tangent Kernel (eNTK) during training. Our key findings reveal a two-phase learning process: i) in Phase I, the eNTK evolves significantly, signaling the rich regime, and ii) in Phase II, the eNTK keeps evolving but is constrained in a narrow space, a phenomenon we term the cone effect. This two-phase framework builds on the hypothesis proposed by Fort et al. (2020), but we uniquely identify the cone effect in Phase II, demonstrating its significant performance advantages over fully linearized training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。