揭示两层神经网络分阶段学习的数学机制，明确各阶段时间阈值。
Singular perturbations and hierarchical learning in two-layer neural networks
- 通过奇异摄动理论分析双层网络梯度流，控制两层训练速度差异。
- 证明常数与线性项在预测时间内被准确恢复，达到精确临界点。
- 发现二次项学习触发权重分布突变，少数神经元显著增长，其余重排维持已学特征。
我们研究高维下无限宽两层神经网络在拟合错误设定单指标模型时的总体梯度流。两层联合优化，通过一个摄动参数调节首层与次层的相对训练速度。该设置由Berthier、Montanari和Zhou在\\[cite{berthier2024learning]提出，他们猜想当次层训练快于首层时会出现分层学习并具有明确的时间尺度。本文证明：隐藏链接函数的常数项与线性项确实在预测时间尺度内被恢复，并达到精确的显式阈值。随后我们分析二次项学习的启动条件，表明前期学习成分对后续动态有本质影响。证明基于关于奇异摄动流的定量逼近结果，其演化靠近由积分约束定义的流形。现象层面，当触及隐藏链接的二次项时，权重的经验测度表现出奇异性：极小比例的神经元显著增长，其余则重组以保持已学成分不变。
原文摘要 · Abstract (English)
We study the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model in high dimension. The two layers are optimized jointly, with a perturbative parameter tuning the relative training speed between the first and second layer. This setting was considered by Berthier, Montanari and Zhou in \cite{berthier2024learning}, who conjectured a hierarchical learning scenario with explicit timescales as the second layer is trained faster than the first. In this paper, we prove that the constant and linear components of the hidden link function are indeed recovered within the predicted timescales, at sharp explicit thresholds. We then analyze the onset of learning of the quadratic component and show that the components learned at earlier stages continue to influence the dynamics in an essential way. Our proof is based on quantitative approximation results for singularly perturbed flows evolving near a manifold defined by integral constraints. At a phenomenological level, we also show that the empirical measure of the weights displays singular behaviour when reaching the quadratic component of the hidden link, with a small fraction of neurons growing significantly while the remaining ones rearrange to preserve the components already learned.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。