Transformer通过分阶段学习稀疏注意力模式，逐步掌握不同历史位置的信息复制能力。
Incremental Learning of Sparse Attention Patterns in Transformers
- 分阶段学习：每阶段专注特定历史位置的稀疏注意力
- 从竞争到协作：各注意力头从争夺重要位置转为分工合作
- 理论揭示无显式课程下结构化注意力与头专化的演化机制
本文研究在高阶马尔可夫链上训练的简单Transformer模型，该任务要求模型整合多个历史位置的信息，且各位置统计重要性不同。我们发现，Transformer以分阶段方式学习此任务，每一阶段对应通过稀疏注意力模式复制部分位置信息。学习动态从竞争阶段（所有头聚焦最显著位置）过渡到协作阶段（各头专注于不同模式）。我们用简化微分方程建模该动态，并证明系统分阶段收敛。功能上，这些阶段构成一系列表达能力递增的误设模型，仅在最后阶段达到完整模型类。整体而言，本研究提供了无显式课程下结构化注意力与头专化的理论解释，对序列任务泛化具有启示。
原文摘要 · Abstract (English)
This paper studies simple transformers trained on a high-order Markov chain, where the model must incorporate information from multiple past positions, each with different statistical importance. We show that transformers learn the task incrementally, with each stage corresponding to learning how to copy information from a subset of positions via a sparse attention pattern. Notably, the learning dynamics transition from a competitive phase, where all heads focus on the statistically most important positions, to a cooperative phase, where different heads specialize in different patterns. We model these dynamics with simplified differential equations and prove stage-wise convergence of the resulting system. Functionally, these stages correspond to a sequence of increasingly expressive misspecified models, with the full model class reached only at the end. Overall, we give a theoretical account of how structured attention patterns and head specialization emerge in stages without an explicit curriculum, with implications for generalization in sequential tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。