通过强化关键注意力头提升长文本检索效果
SEAL: Scaling to Emphasize Attention for Long-Context Retrieval
- 识别并增强与长文本检索相关的特定注意力头
- 在多个任务和模型上显著提升长上下文检索性能
- 适合需要精准长文档理解的场景，如法律、科研
尽管许多先进大模型设计用于处理长序列数据，但在序列长度限制内仍会出现明显的质量下降。本文提出一种新方法 SEAL（Scaling to Emphasize Attention for Long-context retrieval），通过增强大语言模型在长上下文中的检索表现。我们发现某些注意力头与长上下文检索密切相关，其强度与检索得分呈正相关或负相关，调整这些头的权重可大幅提高模型在长上下文中的表现。基于此洞察，我们构建了一个基于生成数据的学习机制，以突出这些关键注意力头。实验表明，SEAL 在多种任务和模型上均实现显著性能提升。此外，结合现有无需训练的上下文扩展技术，SEAL 能在保持输出可靠性的同时，进一步扩展大模型的上下文处理能力。
原文摘要 · Abstract (English)
While many advanced LLMs are designed to handle long sequence data, we can still observe notable quality degradation even within the sequence limit. In this work, we introduce a novel approach called Scaling to Emphasize Attention for Long-context retrieval (SEAL), which enhances the retrieval performance of large language models (LLMs) over long contexts. We observe that specific attention heads are closely tied to long-context retrieval, showing positive or negative correlation with retrieval scores, and adjusting the strength of these heads boosts the quality of LLMs in long context by a large margin. Built on this insight, we propose a learning-based mechanism that leverages generated data to emphasize these heads. By applying SEAL, we achieve significant improvements in long-context retrieval performance across various tasks and models. Additionally, when combined with existing training-free context extension techniques, SEAL extends the contextual limits of LLMs while maintaining highly reliable outputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。