让大模型注意力更聚焦，提升长文本问答能力
MuDAF: Long-Context Multi-Document Attention Focusing through Contrastive Learning on Attention Heads
- 通过对比学习优化注意力头分布，增强关键信息关注
- 在多文档问答中显著提升准确率，减少无关内容干扰
- 适合需要长文本理解与精准检索的场景
大语言模型在处理长上下文时，常因输入中的无关信息导致注意力分散，严重削弱其长程推理能力。受近期关于检索头在长文本事实性判断中有效性的启发，我们提出一种新方法——多文档注意力聚焦（MuDAF），直接通过对比学习在注意力头层面优化注意力分布。实验表明，MuDAF能显著提升大模型在长文本问答任务中的表现，尤其在多文档问答场景下效果突出。对检索得分和注意力可视化结果的广泛评估显示，该方法具备使注意力头更聚焦于相关信息、有效减少注意力分散的巨大潜力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) frequently show distracted attention due to irrelevant information in the input, which severely impairs their long-context capabilities. Inspired by recent studies on the effectiveness of retrieval heads in long-context factutality, we aim at addressing this distraction issue through improving such retrieval heads directly. We propose Multi-Document Attention Focusing (MuDAF), a novel method that explicitly optimizes the attention distribution at the head level through contrastive learning. According to the experimental results, MuDAF can significantly improve the long-context question answering performance of LLMs, especially in multi-document question answering. Extensive evaluations on retrieval scores and attention visualizations show that MuDAF possesses great potential in making attention heads more focused on relevant information and reducing attention distractions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。