发现注意力退化不直接影响上下文召回，关键在功能词的隐状态计算。
Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models
- 通过六组实验验证注意力退化与上下文召回无因果关系。
- 引入句法边界逗号可显著缓解40-80词范围内的预测退化。
- 注意力机制优化无效，模型性能取决于功能词的隐藏表征而非注意力权重。
Transformer可解释性中普遍报告平均跨位置注意力退化，但其是否因果限制上下文检索尚未验证。我们在GPT-2、LLaMA-3.2-1B/3B、OPT-1.3B和distilgpt2上开展六项协同实验。首先刻画5-100词短时注意力退化，发现普遍存在指数衰减后平台期模式，衰减速率与网络深度呈负相关，并呈现架构特异的逐层熵特征。功能词锚定效应具架构依赖性：OPT-1.3B（绝对位置编码）显示距离相关的介词特异性，GPT-2呈现均匀非特异性依赖，而LLaMA（RoPE）在长距离下出现反转。在句法边界插入逗号可因果降低40-80词范围内的预测退化，且效果与句法边界对齐有关，而非词密度。进一步因果测试表明，导向功能词位置的继发感知注意力（RAA）使注意力质量提升16-24%，但在GPT-2和LLaMA-1B上无显著效果，对LLaMA-3B有初步损害，OPT-1.3B表现混合且净效应接近零。多事实检索探测进一步显示，退化速率无法预测各模型的检索准确率。结论：平均注意力退化主要为描述性而非预设性——功能词的贡献源于其隐状态计算，而非接收的注意力权重，这对可解释性方法及基于注意力得分的优化（如KV缓存淘汰）具有深远影响。
原文摘要 · Abstract (English)
Mean cross-positional attention degradation is widely reported in transformer interpretability, yet whether it causally limits contextual retrieval remains untested. We present six coordinated experiments across GPT-2, LLaMA-3.2-1B/3B, OPT-1.3B, and distilgpt2. We first characterise short-term (5-100 token) attention degradation, finding a universal exponential-then-plateau pattern whose rate is inversely correlated with depth, with distinct layer-wise entropy signatures per architecture. Function token anchoring proves architecture-dependent: OPT-1.3B (absolute positional encoding) shows distance-dependent preposition specificity, GPT-2 shows uniform non-specific dependence, and LLaMA (RoPE) shows reversal at long distances. Strategic comma insertion at clause boundaries causally reduces prediction degradation in the 40-80 token range, with the benefit tied to syntactic boundary alignment rather than token density. We then test the mechanism causally: Relay-Aware Attention (RAA), which biases attention logits toward function token positions, verifiably increases attention mass by 16-24% yet yields null effects on GPT-2 and LLaMA-1B, preliminary harm on LLaMA-3B, and a mixed effect on OPT-1.3B that nets to approximately zero. Multi-fact retrieval probes further show that degradation rate does not predict retrieval accuracy across models. We conclude that mean attention degradation is largely descriptive rather than prescriptive: function tokens contribute through what their hidden states compute, not through the attention they receive -- with implications for interpretability methodology and attention-score-based inference optimisations such as KV-cache eviction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。