arXiv:2605.03824cs.CLcs.IR2026-05中稿 · SIGIR 2026, Reprod…

测试复杂查询检索效果,发现现有模型依赖语义捷径而非真正推理。

Reproducing Complex Set-Compositional Information Retrieval

论文配图:Reproducing Complex Set-Compositional Information Retrieval
图 1 · 摘自论文原文
  • 构建新基准LIMIT+,用任意属性约束代替预训练知识判断相关性。
  • 神经检索器在QUEST上召回率超BM25一倍,但到LIMIT+上性能暴跌至0.02以下。
  • 密集模型在复杂组合查询中崩溃,稀疏/词法方法更稳定,适合严谨评估。

复杂信息需求常涉及合取、析取和排除的集合组合查询,但当前检索范式是否真正满足这些约束仍不明确,可能存在依赖语义捷径的问题。本文通过可复现性研究,在QUEST及QUEST+变体上评估主流检索方法与针对性推理模型,并引入LIMIT+这一受控基准,其中相关性由任意属性谓词和约束满足决定,而非预训练知识。结果表明:(i) 在QUEST上,最优神经检索器的召回率@100超过0.41,是BM25(<0.20)的两倍以上,但如ReasonIR和Search-R1等推理目标方法并未持续优于通用检索器;(ii) 在LIMIT+上,性能提升无法迁移,最强的QUEST方法召回率从约0.42骤降至低于0.02,而经典词法检索提升至约0.96;(iii) 按组合深度分层分析显示,所有方法均出现性能下降,其中代数稀疏与词法方法表现更稳定,而密集方法严重崩溃。代码与数据生成脚本已公开,以支持未来可复现研究与受控评估。

原文摘要 · Abstract (English)

Complex information needs may involve set-compositional queries using conjunction, disjunction, and exclusion, yet it remains unclear whether current retrieval paradigms genuinely satisfy such constraints or exploit `semantic shortcuts'. We conduct a reproducibility study to benchmark major retrieval families and reasoning-targeted methods on QUEST and QUEST+Variants, and introduce LIMIT+, a controlled benchmark where relevance depends on arbitrary attribute predicates and constraint satisfaction, and less on pretrained knowledge. Our findings show that (i) on QUEST, the best neural retrievers achieve an effectiveness that is more than double what can be achieved with BM25 (Recall@100 ${>}$0.41 vs.\ 0.20), but reasoning-targeted methods like ReasonIR and Search-R1 do not outperform general-purpose retrievers uniformly; (ii) on LIMIT+, gains fail to transfer, where the strongest QUEST method collapses from Recall@100${\approx}$0.42 to below 0.02, while classic lexical retrieval gains to ${\sim}$0.96. Lastly, (iii) stratifying by compositional depth reveals a consistent degradation across all methods, where algebraic sparse and lexical methods show more stable performance while dense approaches collapse. We release code and LIMIT+ data generation scripts to support future reproducibility and controlled evaluation.

信息检索可复现性复杂查询

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。