发现时间注意力存在对角线信息衰减，提出新方法缓解
Stochastic Parroting in Temporal Attention -- Regulating the Diagonal Sink
- 通过雅可比敏感度分析揭示时间注意力的对角线衰减机制
- 证明长序列下非对角线注意力得分随长度下降，导致早期信息丢失
- 设计正则化方法有效抑制对角线衰减，适合时序建模研究者
时空模型需同时捕捉空间结构与时间动态，但易在时空间发生信息退化。已有研究指出因果注意力或时间卷积存在对首段令牌的过压缩偏差。本文推导了时间注意力层雅可比期望值的敏感度界，理论上证明非对角线注意力得分依赖于序列长度，且时间注意力矩阵存在对角线注意力衰减（diagonal sink）。我们提出正则化方法，并在实验中验证其有效性。
原文摘要 · Abstract (English)
Spatio-temporal models analyze spatial structures and temporal dynamics, which makes them prone to information degeneration among space and time. Prior literature has demonstrated that over-squashing in causal attention or temporal convolutions creates a bias on the first tokens. To analyze whether such a bias is present in temporal attention mechanisms, we derive sensitivity bounds on the expected value of the Jacobian of a temporal attention layer. We theoretically show how off-diagonal attention scores depend on the sequence length, and that temporal attention matrices suffer a diagonal attention sink. We suggest regularization methods, and experimentally demonstrate their effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。