通过归纳偏置视角揭示Transformer中突现学习的机制与调控方式。
Explaining Grokking in Transformers through the Lens of Inductive Bias
- 分析层归一化位置对突现学习速度的影响，揭示其通过注意力熵和捷径学习调节作用。
- 发现优化设置干扰读出尺度作为懒惰训练控制变量的有效性，特征演化呈连续过程。
- 证明泛化能力可预测地由特征可压缩性驱动，适用于多种归纳偏置调制场景。
我们从归纳偏置的角度研究Transformer中的突现学习现象：即网络因架构或优化方式产生的偏好倾向，使其更倾向于选择某一解。首先，我们发现层归一化（LN）的位置显著影响突现学习的速度，该效应可通过特定路径上的LN对捷径学习和注意力熵的调节来解释。其次，我们考察了不同优化设置对突现学习的影响，发现先前提出的读出尺度控制在本设置下可能被学习率和权重衰减所混淆。进一步表明，特征在整个训练过程中持续演化，提示突现学习并非简单的懒惰到丰富学习范式的跃迁。最后，我们展示在多种归纳偏置调制条件下，泛化能力均可由特征可压缩性预测。代码已开源。
原文摘要 · Abstract (English)
We investigate grokking in transformers through the lens of inductive bias: dispositions arising from architecture or optimization that let the network prefer one solution over another. We first show that architectural choices such as the position of Layer Normalization (LN) strongly modulates grokking speed. This modulation is explained by isolating how LN on specific pathways shapes shortcut-learning and attention entropy. Subsequently, we study how different optimization settings modulate grokking, inducing distinct interpretations of previously proposed controls such as readout scale. Particularly, we find that using readout scale as a control for lazy training can be confounded by learning rate and weight decay in our setting. Accordingly, we show that features evolve continuously throughout training, suggesting grokking in transformers can be more nuanced than a lazy-to-rich transition of the learning regime. Finally, we show how generalization predictably emerges with feature compressibility in grokking, across different modulators of inductive bias. Our code is released at https://tinyurl.com/y52u3cad.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。