用动态检索增强稀疏注意力，让长文本模型又快又准。
MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers
- 通过上下文检索动态补充注意力信息，突破固定窗口限制。
- 在合成与真实任务中显著提升稀疏注意力模型性能。
- 适合需要长距离记忆的高效大模型应用，如文档理解。
传统注意力机制的二次计算开销严重制约了大语言模型在长文本场景下的可扩展性与实际部署。现有方法常采用局部注意力窗口等刚性结构约束，但导致需精准长程召回的任务性能大幅下降。本文提出MATCH框架，通过高效检索系统动态融合上下文信息，增强稀疏注意力机制。实验证明，MATCH显著提升了稀疏注意力模型在合成与真实自然语言任务上的表现。结果表明，MATCH是一种通用且高效的方案，可在保持稀疏注意力效率优势的同时，显著增强上下文检索能力。
原文摘要 · Abstract (English)
The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particularly in long-context scenarios. To improve efficiency, existing approaches often enforce rigid structural constraints such as local attention windows. However, these strategies typically lead to substantial performance degradation on tasks requiring precise long-range recall. In this work, we propose MATCH, a scalable and efficient framework that augments sparsified attention mechanisms with dynamically integrated in-context information through an efficient retrieval system. Empirical results show that MATCH significantly improves the performance of sparse-attention models on both synthetic and real-world natural-language tasks. These findings highlight the versatility of MATCH as a general approach for enhancing in-context retrieval capabilities while maintaining the efficiency benefits of sparse attention architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。