测试文档中隐含事实的推理检索能力，发现现有模型表现极差。
ImpliRet: Benchmarking the Implicit Fact Retrieval Challenge
- 查询简单但需理解文档中的时间、算术和常识关系。
- 最佳模型nDCG@10仅14.91%，远低于常规检索水平。
- 长文本模型也难胜任，适合评估复杂推理检索系统。
检索系统在许多自然语言处理流程中至关重要，但通常依赖关键词重叠和词义相似性等表面线索。为评估超越这些浅层信号的检索能力，近期基准引入了需要推理的查询；然而，它们主要将挑战转移至查询端处理技术（如提示工程或多跳检索）。相比之下，我们提出ImpliRet，将推理挑战转移到文档端：查询简洁，但相关性依赖于文档中通过时间关系（如“两天前”）、算术关系和世界知识所隐含的事实。我们评估了多种稀疏与密集检索器，所有模型均表现不佳，最佳nDCG@10仅为14.91%。我们还测试了长上下文模型是否能克服此局限，但在仅30篇文档（包含正例）的短上下文中，GPT-o4-mini得分也仅55.54%，表明文档端推理仍是重大挑战。代码已开源。
原文摘要 · Abstract (English)
Retrieval systems are central to many NLP pipelines, but often rely on surface-level cues such as keyword overlap and lexical semantic similarity. To evaluate retrieval beyond these shallow signals, recent benchmarks introduce reasoning-heavy queries; however, they primarily shift the burden to query-side processing techniques -- like prompting or multi-hop retrieval -- that can help resolve complexity. In contrast, we present Impliret, a benchmark that shifts the reasoning challenge to document-side processing: The queries are simple, but relevance depends on facts stated implicitly in documents through temporal (e.g., resolving "two days ago"), arithmetic, and world knowledge relationships. We evaluate a range of sparse and dense retrievers, all of which struggle in this setting: the best nDCG@10 is only 14.91%. We also test whether long-context models can overcome this limitation. But even with a short context of only thirty documents, including the positive document, GPT-o4-mini scores only 55.54%, showing that document-side reasoning remains a challenge. Our codes are available at github.com/ZeinabTaghavi/IMPLIRET.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。