首个测试长篇叙事中多跳推理的基准,揭示大模型在复杂推理中的系统性失败。
NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts
- 构建基于关键词的链式推理框架,确保多跳答案与故事情节一致
- 在64k-128k-token小说片段上测试,模型准确率随跳跃次数和上下文长度上升显著下降
- 提供可复现的诊断环境,适合评估大模型在长文本推理中的鲁棒性
当前大型语言模型难以回答跨越数万词的长文本问题,尤其涉及多跳推理时。现有基准仅单独考察长上下文理解或多跳推理,缺乏在自然叙事场景中联合变化上下文长度与推理深度的评估。我们提出NovelHopQA,首个在83部公共领域小说的64k-128k-token片段上评估1-4跳问答的基准。通过关键词引导的流水线构建符合连贯叙事的分步推理链。评估7个前沿模型,并采用真值上下文过滤确保问题可解。人工标注验证答案对齐性与跳数深度。额外引入检索增强生成(RAG)评估,测试仅提供部分段落时的表现。发现即使顶尖模型,准确率也随跳数和上下文增长持续下降,表明规模并非推理鲁棒性的保障。故障模式分析揭示常见问题包括最终跳信息整合缺失与长距离推理漂移。NovelHopQA为大规模多跳推理提供了可控诊断环境。代码与数据集已开源。
原文摘要 · Abstract (English)
Current large language models (LLMs) struggle to answer questions that span tens of thousands of tokens, especially when multi-hop reasoning is involved. While prior benchmarks explore long-context comprehension or multi-hop reasoning in isolation, none jointly vary context length and reasoning depth in natural narrative settings. We introduce NovelHopQA, the first benchmark to evaluate 1-4 hop QA over 64k-128k-token excerpts from 83 full-length public-domain novels. A keyword-guided pipeline builds hop-separated chains grounded in coherent storylines. We evaluate seven state-of-the-art models and apply oracle-context filtering to ensure all questions are genuinely answerable. Human annotators validate both alignment and hop depth. We additionally present retrieval-augmented generation (RAG) evaluations to test model performance when only selected passages are provided instead of the full context. We noticed consistent accuracy drops with increased hops and context length increase, even for frontier models-revealing that sheer scale does not guarantee robust reasoning. Failure-mode analysis highlights common breakdowns such as missed final-hop integration and long-range drift. NovelHopQA offers a controlled diagnostic setting to test multi-hop reasoning at scale. All code and datasets are available at https://novelhopqa.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。