通过重复上下文提升大模型多跳推理能力
Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context
- 重复呈现上下文，引导模型按最优顺序推理
- 多跳问答任务F1提升30个百分点，合成任务准确率提高70%
- 有效缓解大模型‘中间遗忘’问题，适合检索+思维链场景
多跳推理要求模型基于给定上下文中的支持文档进行多步推理，对大语言模型（LLMs）仍具挑战性。模型常难以过滤无关文档，且性能对支持文档在上下文中的绝对位置敏感。本文识别出另一关键问题：模型性能也受支持文档相对顺序的影响，称为‘顺序错乱上下文问题’。为此，基于理论分析，提出简单有效的上下文重复（CoRe）方法，通过反复提示模型呈现上下文，确保支持文档中某些连续推理片段以最优顺序出现，从而有效引导模型推理方向。应用CoRe后，在多跳问答任务上F1分数最高提升30个百分点，在合成任务上准确率最高提升70个百分点。此外，CoRe有助于缓解大模型著名的‘中间遗忘’问题，并可与基于检索的链式思维（CoT）推理方法有效结合。
原文摘要 · Abstract (English)
Multi-hop reasoning, which requires multi-step reasoning based on the supporting documents within a given context, remains challenging for large language models (LLMs). LLMs often struggle to filter out irrelevant documents within the context, and their performance is sensitive to the absolute position of supporting documents within that context. In this paper, we identify an additional challenge: LLMs' performance is also sensitive to the order, relative position, in which the supporting documents are presented. We refer to this as the misordered context problem. To address this issue, based on the theoretical approach, we propose a simple yet effective method called context repetition (CoRe), which involves prompting the model by repeatedly presenting the context. This ensures that certain contiguous reasoning segments within supporting documents are presented in the optimal order, effectively guiding the model's reasoning in the appropriate direction. Applying CoRe, we improve the F1 score by up to 30%p on multi-hop QA tasks and increase accuracy by up to 70%p on a synthetic task. Additionally, CoRe helps mitigate the well-known "lost-in-the-middle" problem in LLMs and can be effectively combined with retrieval-based approaches utilizing Chain-of-Thought (CoT) reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。