arXiv:2605.18857cs.IRcs.AI2026-05中稿 · ICLR被引 2

高召回率未必高效,该用新指标判断检索是否真有选择性。

The 99% Success Paradox: When Near-Perfect Retrieval Equals Random Selection

论文配图:The 99% Success Paradox: When Near-Perfect Retrieval Equals Random Selection
图 1 · 摘自论文原文
  • 提出BoR指标,校正随机基线,揭示高成功率下的随机选择本质。
  • 在20 Newsgroups上,K=100时成功率超99%但选择性近乎为零。
  • 适用于评估大模型检索、工具选择等场景,提醒避免无效深度计算。

信息检索系统长期面向人类用户设计,依赖人工过滤冗余信息,因此侧重提升相关文档的召回与排序,而非保持结果简洁。然而大语言模型(LLM)缺乏此类过滤能力,导致高召回率可能掩盖随机选择的本质。本文引入比特-随机比(Bits-over-Random, BoR)作为选择性度量:$BoR = \log_{2}\left(\frac{\mathrm{P}_{obs}}{\mathrm{P}_{rand}}\right)$,其中$\mathrm{P}_{rand}$为超几何分布基线(以覆盖率为标准,即前K位至少一个相关文档)。在20 Newsgroups数据集上,BM25与SPLADE在K=100时覆盖率均超过99%,但BoR≈0,表明此时选择性已趋近随机。当预期覆盖率比$\left(\frac{K \cdot \bar{R}_{q}}{N}\right)$超过3-5时,基线主导,选择性崩溃。下游检索增强生成(RAG)实验验证此现象:当K=100时LLM准确率显著下降,与近零的BoR上限一致。在BEIR/SciFact及MS MARCO上,41个系统虽存在13个百分点的召回差距,却均接近理论上限(差值<0.2比特),说明该基准具有跨稀疏与大规模场景普适性。进一步发现,该崩溃边界同样适用于大模型代理的工具选择任务,即使选择器完美,小工具库也会导致选择性消失。研究建议在报告传统指标的同时加入BoR,并重新审视检索深度,避免计算开销增加而无实质收益。

原文摘要 · Abstract (English)

For most of the history of information retrieval (IR), search results were designed for human consumers who could scan, filter, and discard irrelevant information on their own. This shaped retrieval systems to optimize for finding and ranking more relevant documents, but not keeping results clean and minimal, as the human was the final filter. However, LLMs have changed that by lacking this filtering ability. To address this, we introduce Bits-over-Random (BoR), a chance-corrected measure of retrieval selectivity that reveals when high success rates mask random-level performance. We measure selectivity as $BoR = \log_{2}\left(\frac{\mathrm{P}_{obs}}{\mathrm{P}_{rand}}\right)$, where $\mathrm{P}_{rand}$ is the hypergeometric baseline for the chosen success rule (here, coverage: $ \geq1 $ relevant in top-$K$). On the 20 Newsgroups dataset, BM25 and SPLADE both report $>99$% success at $K=100$ (coverage), yet $BoR \approx 0$, indicating random-level selectivity at that depth. When the expected coverage ratio $\left(\frac{K \cdot \bar{R}_{q}}{N}\right)$ exceeds 3-5, the baseline dominates and selectivity collapses. Downstream retrieval-augmented generation (RAG) evaluation confirms this pattern: LLM accuracy can degrade substantially at $K=100$, consistent with the near-zero BoR ceiling. In contrast, BoR remains positive on BEIR/SciFact and on MS MARCO (where 41 systems cluster within 0.2 bits of the theoretical ceiling despite a 13-point recall gap), confirming baseline predictions across sparse and large-scale settings. We further show that the collapse boundary applies to LLM agent tool selection, where small catalog sizes cause selectivity to vanish even with perfect selectors. These findings suggest reporting BoR alongside traditional metrics and reconsidering depth choices when additional retrieval provides negligible selectivity gains while inflating computational costs.

检索评估大模型选择性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。