提出可泛化长文本的注意力机制，解决短上下文训练后长上下文推理效果差的问题。
Scale-invariant Attention
- 通过位置相关变换优化注意力得分，实现尺度不变的总注意力与稀疏性。
- 在零样本泛化测试中，长上下文验证损失显著降低。
- 适合需要处理超长文本的场景，如文档检索与生成。
大语言模型研究中一个长期挑战是如何使注意力机制从短上下文训练有效泛化到长上下文推理。我们提出两个理想长上下文注意力机制应满足的条件：尺度不变的总注意力和尺度不变的注意力稀疏性。在高斯假设下，仅需对注意力分数进行位置相关的简单变换即可满足这两个条件。实验表明，所提出的尺度不变注意力方案在零样本泛化任务中显著降低长上下文验证损失，并在长文本检索中表现优异。
原文摘要 · Abstract (English)
One persistent challenge in LLM research is the development of attention mechanisms that are able to generalise from training on shorter contexts to inference on longer contexts. We propose two conditions that we expect all effective long context attention mechanisms to have: scale-invariant total attention, and scale-invariant attention sparsity. Under a Gaussian assumption, we show that a simple position-dependent transformation of the attention logits is sufficient for these conditions to hold. Experimentally we find that the resulting scale-invariant attention scheme gives considerable benefits in terms of validation loss when zero-shot generalising from training on short contexts to validation on longer contexts, and is effective at long-context retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。