神经网络为何迟迟不放弃捷径？答案是参数范数的层级迁移过程。
Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcuts
- 通过参数范数层级渐进转移解释延迟学习现象
- 过渡时间与捷径与结构化表示的范数比值对数相关
- 适用于理解模型何时发现真实特征，适合研究训练动态者
神经网络在训练初期常依赖虚假捷径，经过多轮迭代后才逐渐发现结构化表征。然而，这一转变发生的时间及其可预测性尚不明确。已有研究表明梯度下降倾向于收敛到低范数解，并存在简化偏好，但均无法解释该转变的时间尺度。本文提出范数层级过渡（Norm-Hierarchy Transition, NHT）框架，将延迟表征学习归因于正则化优化中参数范数层级的缓慢遍历过程。当存在多个可插值解且范数不同时，权重衰减会逐步引导模型从高范数的捷径解向低范数的结构化表征移动。我们推导出一个紧致上界，表明过渡延迟随捷径与结构化表示范数比值的对数增长。在模运算、含虚假特征的CIFAR-10、CelebA和Waterbirds数据集上的实验验证了该框架的预测。结果表明，沟谷现象、捷径学习和延迟特征发现均源于训练中范数层级的共同机制。
原文摘要 · Abstract (English)
Neural networks often rely on spurious shortcuts for many epochs before discovering structured representations. However, the mechanism governing when this transition occurs and whether its timing can be predicted remains unclear. Prior work shows that gradient descent converges to low norm solutions and that neural networks exhibit simplicity bias, but neither explains the timescale of the transition from shortcut features to structured representations. We introduce the Norm-Hierarchy Transition (NHT) framework, which explains delayed representation learning as the slow traversal of a hierarchy of parameter norms during regularized optimization. When multiple interpolating solutions exist with different norms, weight decay gradually moves the model from high norm shortcut solutions toward lower norm structured representations. We derive a tight bound showing that the transition delay grows logarithmically with the ratio between shortcut and structured norms. Experiments on modular arithmetic, CIFAR-10 with spurious features, CelebA, and Waterbirds support the predictions of the framework. The results suggest that grokking, shortcut learning, and delayed feature discovery arise from a common mechanism based on norm hierarchy traversal during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。