提出查询导向的二次擦除机制，显著提升长序列线性注意力的检索能力。
The Query Knows What to Forget: A Second Erase Direction for Linear Attention
- 引入查询衍生的擦除方向，与键正交，增强状态更新精度。
- 在超过训练窗口长度时仍保持良好检索效果，S-NIAH-1上可用上下文翻倍。
- 适用于需要长序列建模的场景，如超长文本生成与理解。
线性注意力通过固定大小的状态存储信息。在长序列场景下，多个存储项共用同一状态，相互干扰导致检索性能下降。现有基于增量规则的模型（如GDN-2）均从当前词元的键中推导擦除向量，但其读取过程中的干扰由查询衡量，而擦除步骤无法触及查询方向。本文提出查询衍生的擦除方向（QED），增加一个与键正交的第二擦除方向。从快速权重视角看，键引导的增量修改无法改变读取状态中与键正交的部分；而该部分恰好用于抵消沿查询方向测量的旧状态内容。实验表明，该方法在所有超过训练窗口长度的序列上均提升检索性能，并在S-NIAH-1数据集上将有效上下文长度约翻倍。
原文摘要 · Abstract (English)
Linear attention keeps a state of fixed size. At long context, many stored items share this state, and interference between them degrades retrieval. Gated DeltaNet-2 (GDN-2), like every delta-rule model before it, derives its erase vector from the key of the current token. However, the interference in its reads is measured through the query, and the erase step cannot reach it. We introduce the Query-derived Erase Direction (QED). QED adds a second erase direction derived from the query and orthogonal to the key. In the fast-weight view, a key-directed delta edit cannot change the key-orthogonal part of a read. It uses the editable part to cancel old-state content measured along the query. It also improves retrieval at every length past the training window, and it about doubles the usable context length on S-NIAH-1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。