改进注意力机制，让模型更擅长处理超长序列。
TRA: Better Length Generalisation with Threshold Relative Attention
- 用选择性稀疏过滤无关信息，避免干扰注意力计算。
- 引入上下文相关相对距离，提升长序列泛化能力。
- 适合研究长文本生成与模型泛化性的研究人员。
Transformer 在长度泛化方面表现不佳，即使在基础任务上也是如此。我们检验了自注意力机制的两个关键缺陷：一是无法完全消除无关信息；二是位置偏差可能导致即使键值点积为负（即不相关），仍被意外加权——当距离超出训练分布时尤为危险。这两类问题导致泛化困难叠加。我们通过结合 a）选择性稀疏——从注意力 softmax 中彻底移除无关键；b）上下文感知的相对距离——仅考虑对查询重要的键之间的距离，来缓解这些问题。结果表明，重构注意力机制后，仅解码器型 Transformer 的泛化能力得到显著提升。
原文摘要 · Abstract (English)
Transformers struggle with length generalisation, displaying poor performance even on basic tasks. We test whether these limitations can be explained through two key failures of the self-attention mechanism. The first is the inability to fully remove irrelevant information. The second is tied to position, even if the dot product between a key and query is highly negative (i.e. an irrelevant key) learned positional biases may unintentionally up-weight such information - dangerous when distances become out of distribution. Put together, these two failure cases lead to compounding generalisation difficulties. We test whether they can be mitigated through the combination of a) selective sparsity - completely removing irrelevant keys from the attention softmax and b) contextualised relative distance - distance is only considered as between the query and the keys that matter. We show how refactoring the attention mechanism with these two mitigations in place can substantially improve the generalisation capabilities of decoder only transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。