长文本问答中模型易忽略中间信息，本文提出改进方法提升多跳推理能力。
Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA
- 通过知识图谱提取与摘要压缩减少冗余内容
- 链式思维提示显著提升跨段落信息整合能力
- 解决长文本中信息位置依赖问题，适合多跳问答场景
先前研究发现，近期的长上下文语言模型未能均衡利用输入中间部分的信息，更偏好尾部内容，导致在需要全面使用输入各部分时产生偏差。现有工作主要关注单个关键信息的情况，但当多个必要信息分散在长文档中时，问题更为复杂。本文在多跳问答任务中揭示了‘中间迷失’现象：性能不仅随信息距边缘距离下降，也受信息间距离影响。通过知识图谱三元组抽取和摘要压缩减少冗余，并结合链式思维提示，模型对分散信息的整合能力显著提升。实验表明，在LongChat、MMLU-long等数据集上，该方法有效缓解了信息位置偏差问题。
原文摘要 · Abstract (English)
Previous work finds that recent long-context language models fail to make equal use of information in the middle of their inputs, preferring pieces of information located at the tail ends which creates an undue bias in situations where we would like models to be equally capable of using different parts of the input. Thus far, the problem has mainly only been considered in settings with single pieces of critical information, leading us to question what happens when multiple necessary pieces of information are spread out over the inputs. Here, we demonstrate the effects of the "lost in the middle" problem in the multi-hop question answering setting -- in which multiple reasoning "hops" over disconnected documents are required -- and show that performance degrades not only with respect to the distance of information from the edges of the context, but also between pieces of information. Additionally, we experiment with means of alleviating the problem by reducing superfluous document contents through knowledge graph triple extraction and summarization, and prompting models to reason more thoroughly using chain-of-thought prompting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。