揭示语言模型中归纳头出现的触发条件，可提前预测其形成时机。
Predicting the Emergence of Induction Heads in Language Model Pretraining
- 用批量大小与上下文长度的乘积预测归纳头出现时间。
- 重复性高且可靠的二元组是归纳头形成的决定性因素。
- 局部依赖性足够引发归纳头，适合研究模型内部机制者阅读。
被称为归纳头（IHs）的特殊注意力头被认为支撑了现代语言模型的上下文学习能力；然而，其在语言建模中的确切形成机制仍不清楚。本研究探讨了训练数据的统计特性与归纳头生成之间的关系，涵盖自然和合成数据设置。结果表明：（1）一个结合批次大小与上下文大小的简单公式可准确预测归纳头的形成点，且该点与模型规模无关；（2）表面二元组重复频率与可靠性显著影响归纳头形成，我们发现二者存在有效的决策边界；（3）高重复频率与可靠性的局部依赖足以促成归纳头，而类别性和边际分布形状则在决策边界附近调节其形成过程。
原文摘要 · Abstract (English)
Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern language models; yet, a precise characterization of their emergence, especially in the context of language modeling, remains wanting. In this study, we investigate the relationship between statistical properties of the training data and IH formation in both natural and synthetic training data settings. We show that: (1) a simple equation combining batch size and context size predicts the point at which IHs form and that this emergence point is agnostic to model size; (2) surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find an effective decision boundary in terms of these two values; (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but categoriality and the shape of the marginal distribution appear to modulate IH formation near the decision boundary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。