揭示了Transformer位置偏见的结构根源，解释为何中间信息易被忽略。
A Structural Theory of Position Bias in Transformers
- 基于残差感知的注意力传播理论，分析架构如何导致位置偏好。
- 有限深度下出现U型影响分布，中间位置影响力显著降低。
- 理论预测与预训练模型实测结果高度吻合，适用于语言模型优化。
Transformer模型存在系统性位置偏好，但其架构成因仍不明确。该现象与‘信息丢失于中段’（Lost-in-the-Middle）密切相关。本文证明，因果Transformer架构本身即可引发此类行为。通过构建基于残差感知的累积注意力传播结构理论，发现有限深度时，因果掩码与残差连接共同导致广泛且常呈U型的影响分布；无限深度下，该框架解决了以往仅关注注意力的坍缩理论与实际模型表现之间的矛盾：残差连接从根本上改变了累积注意力动态。实验表明，理论预测的影响分布与预训练语言模型中实际输入词元影响力高度一致。
原文摘要 · Abstract (English)
Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood. This bias is closely connected to the Lost-in-the-Middle phenomenon, where models underutilize information placed in the middle of the context. We show that Lost-in-the-Middle-type behavior can arise from the architecture of causal Transformers itself. To do so, we develop a structural theory of position bias based on residual-aware cumulative attention rollout. At finite depth, causal masking and residual connections induce broad, often U-shaped, influence profiles. At infinite depth, our framework resolves a discrepancy between prior attention-only collapse theory and practical Transformer behavior: residual connections fundamentally change cumulative attention dynamics. Empirically, the predicted profiles closely match measured input-token influence in pretrained language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。