arXiv:2510.12251cs.CL2025-10NeurIPS被引 8

提出可即插即用的注意力优化框架,提升大模型处理长文档问答的能力。

DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answering

  • 通过分层注意力追踪与位置加权缓解中间信息丢失问题
  • 抑制关键与无关段落间的信息干扰,强化长距离语义关联
  • 无需修改模型结构或训练参数,适配主流大模型

尽管大语言模型在多个领域展现巨大潜力,但在多文档问答任务中仍存在明显局限。首先,长距离依赖建模困难,模型难以聚焦长文本中的关键信息,削弱重要语义连接;其次,多数模型存在“中间信息丢失”问题,难以有效处理长输入中间部分的内容。现有解决方案或截断全局依赖,或需昂贵微调,缺乏通用且简洁的解决路径。为此,本文提出双阶段自适应锐化(DSAS)框架,包含两个模块:(i) 上下文门控加权(CGW)模块通过分层注意力追踪与位置感知加权,缓解“中间信息丢失”;(ii) 互斥注意力抑制(RAS)模块通过抑制关键与无关文本间的信息交互,增强对关键段落的关注,缓解长距离依赖建模缺陷。DSAS作为即插即用方案,无需模型架构修改或额外训练参数。在四个基准上的实验表明,其在主流大模型(Llama、Qwen、Mistral、Deepseek)上均有效,尤其在Llama-3.1-8B-Instruct和Qwen2.5-14B-Instruct上平均F1得分提升4.2%。消融实验证实CGW与RAS模块均具关键贡献。附录进一步验证了其鲁棒性与可扩展性。

原文摘要 · Abstract (English)

While large language models (LLMs) show considerable promise across various fields, they have notable limitations in handling multi-document question answering (Multi-doc QA) tasks. The first challenge is long-range dependency modeling, where LLMs struggle to focus on key information in long texts, which weakens important semantic connections. Second, most LLMs suffer from the ''lost-in-the-middle'' issue, where they have difficulty processing information in the middle of long inputs. Current solutions either truncate global dependencies or demand costly finetuning, ultimately lacking a universal and simple solution for these challenges. To resolve these limitations, we propose Dual-Stage Adaptive Sharpening (DSAS) containing two modules. (i) The Contextual Gate Weighting (CGW) module alleviates ''lost-in-the-middle'' by assessing paragraph relevance through layer-wise attention tracking and position-aware weighting. (ii) The Reciprocal Attention Suppression (RAS) module enhances focus on critical paragraphs by suppressing information exchange between key and irrelevant texts, thus mitigating the limitations in long-range dependency modeling. Notably, DSAS functions as a plug-and-play solution requiring no architectural modifications or extra training parameters. Extensive experiments on four benchmarks demonstrate DSAS's efficacy across mainstream LLMs (Llama, Qwen, Mistral, and Deepseek), with an average F1-score improvement of 4.2% in Multi-doc QA tasks on Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct. Ablation studies confirm the essential contributions of both the CGW and RAS modules. In addition, detailed discussions in the Appendix further validate the robustness and scalability of DSAS.

多文档问答注意力机制大模型优化即插即用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。