预条件梯度下降可缓解神经网络学习偏见，加速模型收敛。
On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regime
- 使用高斯-牛顿等预条件优化方法，打破频率学习偏见。
- 实验验证预条件梯度下降能显著缩短模型达到泛化所需的训练时间。
- 适合关注优化算法与深度学习动态机制的研究者阅读。
谱偏差指神经网络优先学习低频特征，虽利于泛化但阻碍精细结构捕捉。延迟泛化现象（即grokking）是训练速度慢的重要原因，有假说认为其源于从NTK惰性区向丰富特征区的过渡。本文研究预条件梯度下降（PGD，如高斯-牛顿法）对谱偏差和grokking的影响。理论与实证表明，PGD可缓解谱偏差问题。基于丰富的学习区假设，我们发现PGD能减少grokking延迟：在无谱偏差阻碍下，参数空间探索趋于均匀。实验结果支持该预测，证实grokking是惰性区（由NTK描述）与丰富区之间的过渡行为。这些发现深化了对优化动态、谱偏差与学习阶段间相互作用的理解。
原文摘要 · Abstract (English)
Spectral bias, the tendency of neural networks to learn low frequencies first, can be both a blessing and a curse. While it enhances the generalization capabilities by suppressing high-frequency noise, it can be a limitation in scientific tasks that require capturing fine-scale structures. The delayed generalization phenomenon known as grokking is another barrier to rapid training of neural networks. Grokking has been hypothesized to arise as learning transitions from the NTK to the feature-rich regime. This paper explores the impact of preconditioned gradient descent (PGD), such as Gauss-Newton, on spectral bias and grokking phenomena. We demonstrate through theoretical and empirical results how PGD can mitigate issues associated with spectral bias. Additionally, building on the rich learning regime grokking hypothesis, we study how PGD can be used to reduce delays associated with grokking. Our conjecture is that PGD, without the impediment of spectral bias, enables uniform exploration of the parameter space in the NTK regime. Our experimental results confirm this prediction, providing strong evidence that grokking represents a transitional behavior between the lazy regime characterized by the NTK and the rich regime. These findings deepen our understanding of the interplay between optimization dynamics, spectral bias, and the phases of neural network learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。