用可学习稀疏注意力提升长序列建模能力，解决注意力分散问题。
Long-Context Generalization with Sparse Attention
- 采用可学习温度的稀疏注意力机制，动态控制关注模式
- 在合成任务上实现1000倍长度外推，语言建模长上下文性能更优
- 适合需要精准定位固定模式的长文本任务
基于Transformer的架构传统上使用softmax计算注意力权重，生成对所有词元的稠密分布。尽管在许多场景中有效，但这种稠密性会损害需要精确聚焦固定模式的任务：随着序列变长，无关词元累积注意力概率，导致注意力分散和表征坍塌。本文表明，使用α-entmax的动态稀疏注意力机制可避免这些问题，因其能为无关词元分配零值。此外，我们提出自适应可扩展的Entmax（ASEntmax），为α-entmax引入可学习温度参数，使注意力分布可在稀疏（聚焦模式）与稠密（类似softmax）之间插值。在合成任务和语言建模上的实证评估显示，ASEntmax显著优于softmax、可扩展softmax及固定温度α-entmax基线，在合成基准上达到最高1000×长度外推，并在语言建模中实现更优的长上下文泛化能力，同时保持短上下文性能，包括在8倍训练长度下的更好困惑度趋势和更高检索准确率。
原文摘要 · Abstract (English)
Transformer-based architectures traditionally employ softmax to compute attention weights, which produces dense distributions over all tokens in a sequence. While effective in many settings, this density has been shown to be detrimental for tasks that demand precise focus on fixed-size patterns: as sequence length increases, non-informative tokens accumulate attention probability mass, leading to dispersion and representational collapse. We show in this paper that dynamically sparse attention mechanisms using $α$-entmax can avoid these issues, due to their ability to assign exact zeros to irrelevant tokens. Furthermore, we introduce Adaptive-Scalable Entmax (ASEntmax), which endows $α$-entmax with a learnable temperature parameter, allowing the attention distribution to interpolate between sparse (pattern-focused) and dense (softmax-like) regimes. Our empirical evaluation on synthetic tasks and language modeling demonstrates that ASEntmax substantially outperforms softmax, scalable softmax, and fixed-temperature $α$-entmax baselines, achieving up to 1000$\times$ length extrapolation on synthetic benchmarks and superior long-context generalization on language modeling while preserving short-context performance, including better perplexity trends and higher retrieval accuracies at 8$\times$ training length.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。