解释Transformer中'中间遗忘'现象的数学原理
Kinetic theory for Transformers and the lost-in-the-middle phenomenon
- 将自注意力机制建模为非交换粒子系统，用累积量展开分析
- 证明了均场极限并揭示了相关性的下一阶特征，出现U形检索曲线
- 首次严格推导出中间位置令牌最弱的理论依据，适合模型研究者
我们研究因果自注意力动力学——解码器Transformer的简化模型——将其视为非交换相互作用粒子系统。通过将累积量展开方法适配至模型的三角形因果依赖结构，并利用Glauber微积分的非分层方法估计相关性，我们证明了定量均场极限结果及相关性的下一阶刻画。对于独立同分布的均匀分布标记，极限相关性方程可解析求解，从而获得对经验观察到的‘中间遗忘’现象的严格解释：作为提示中源位置函数的标记检索轮廓呈U形，具有首因效应、近因效应，且在显式小参数条件下存在唯一内部最小值。
原文摘要 · Abstract (English)
We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed \emph{lost-in-the-middle} phenomenon: the token retrieval profile, as a function of the source position in the prompt, is $\mathsf{U}$-shaped, with primacy, recency, and a unique interior minimum under an explicit smallness condition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。