发现大模型多跳问答失败源于位置偏见,最弱证据决定整体表现。
Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneck

- 设计注意力引导方法,分离定位与整合失败机制。
- 在18文档3分组设置下,表现受最不可见证据拖累,与绝对位置相关。
- 思维模型可克服噪声,实现接近黄金标准的推理能力。
尽管大规模语言模型(LLMs)具备超长上下文处理能力,但在多跳推理中仍因固有的位置偏见而表现不佳,导致对特定位置信息的忽略。这种失败是由于无法识别证据(识别失败)还是无法整合证据(合成失败)尚不明确。本文提出多焦点注意力指令(MFAI),通过显式引导注意力至特定位置,以解耦上述机制。在两个多跳问答任务(MuSiQue 和 NeoQA)上,针对5个LLM进行实验,发现“最弱环节效应”:在18文档、3分组设置下,多跳推理性能降至最不可见证据的水平,且由绝对位置决定而非事实间的线性距离。匹配的MFAI可缓解识别瓶颈,使低可见度位置准确率提升最高达11.49%;但误导性MFAI则产生不同影响,实体中心任务因垂直推理链更易受影响,事件中心任务因水平证据结构更具鲁棒性。最后,采用系统2推理的思维模型能有效定位并整合所需信息,在嘈杂长上下文环境中达到仅使用黄金标签的基准表现。补充实验在2WikiMultiHopQA、扩展至3-4跳及32B模型上验证了结果在数据集、推理深度和模型规模上的泛化性。
原文摘要 · Abstract (English)
Despite scaling to massive context windows, Large Language Models (LLMs) struggle with multi-hop reasoning due to inherent position bias, which causes them to overlook information at certain positions. Whether these failures stem from an inability to locate evidence (recognition failure) or integrate it (synthesis failure) is unclear. We introduce Multi-Focus Attention Instruction (MFAI), a semantic probe to disentangle these mechanisms by explicitly steering attention towards selected positions. Across 5 LLMs on two multi-hop QA tasks (MuSiQue and NeoQA), we identify the "Weakest Link Effect": in our 18-document, 3-bucket setting, multi-hop reasoning performance collapses to the level of the least visible evidence, governed by absolute position rather than the linear distance between facts. While matched MFAI resolves recognition bottlenecks, improving accuracy by up to 11.49% in low-visibility positions, misleading MFAI yields divergent effects modulated by task topology: entity-centric tasks with vertical reasoning chains are vulnerable, whereas event-centric tasks with horizontal evidence structures are more resilient. Finally, we demonstrate that thinking models utilizing System-2 reasoning effectively locate and integrate the required information, matching gold-only baselines even in noisy, long-context settings. Supplementary experiments on 2WikiMultiHopQA, extended 3-4 hop counts, and a 32B model confirm these findings generalize across datasets, reasoning depths, and model scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。