静态相关性无法预测智能体搜索中的因果作用,关键文档常被误判为无用。
Bridge Evidence: Static Retrieval Utility Does Not Predict Causal Utility in Multi-Step Agentic Search
- 通过反事实重演实验,量化文档在多步搜索中对后续行为的影响
- 仅1/3的阅读文档具因果重要性,多数被静态评估忽略
- 真正关键的是提供能引导下一步搜索的区分性信息实体
检索系统通常以静态方式评估文档有用性:给定问题和文档,看答案是否改善并打分。这一标准在单次阅读时有效,但在语言模型作为搜索代理、需多轮查询与推理时失效,因文档的价值在于其推动后续行动的能力,而非直接回答当前问题。我们通过在HotpotQA上使用ReAct风格代理重现1000个开发集问题,对每个被读取的文档进行删除并重跑后续轨迹,对比原始与反事实结果,计算出三种差异:最终答案质量、下一轮查询检索质量、对话轮次数,由此得到反事实轨迹效用(CTU)得分。将CTU与静态检索增强生成效用(SRU)在23,322个文档观测中交叉分析,两者近乎统计独立(斯皮尔曼等级相关系数ρ = -0.026)。约三分之一的阅读文档具有因果重要性,虽对静态读者看似无用,我们称之为桥接文档。该现象在替换读者为BM25或交叉编码器代理后仍成立,桥接比例达27.2%。第二项实验表明,来自相关候选的可观察实体在代理下一轮查询中出现频率是仅见于无关文档实体的4.02倍(6.1% vs 1.5%,n=227,139),说明桥接文档的核心价值在于提供能引导搜索方向的区分性实体。静态相关性与因果实用性在代理检索中本质不同,优化前者无法带来后者。
原文摘要 · Abstract (English)
Retrieval systems are trained and evaluated on a static idea of usefulness: hand a document and a question to a reader model, see whether the answer improves, and score the document accordingly. The idea holds up when a document is read on its own. It breaks when a language model works as a search agent, issuing several queries and reasoning across turns, because a document can matter for what it lets the agent do next rather than for what it says about the current question. We measure that gap rather than argue it. Using a ReAct style agent over HotpotQA, we replay 1000 development questions and, for every document the agent read, delete it and re-run the rest of the trajectory from that point. Comparing the original run against its counterfactual gives a Counterfactual Trajectory Utility (CTU) score from three deltas: final answer quality, next query retrieval quality, and turn count. Crossing CTU against Static RAG Utility (SRU) over 23,322 document observations, the two are close to statistically independent (Spearman rho = -0.026). Roughly a third of the documents the agent reads are causally load bearing while looking useless to a static reader; we call these bridge documents. The pattern survives when the reader based axis is swapped for a BM25 and cross encoder proxy, giving a bridge cell of 27.2% on an evenly spread axis. A second experiment pins down the mechanism. Using the Observable Entity Relevance (OER) measure from prior work, entities that discriminate relevant from non-relevant candidates appear in the agent's next query 4.02 times more often than entities found only in non-relevant documents (6.1% vs 1.5%, n = 227,139). A bridge document earns its keep by handing the agent a discriminative entity that redirects the search. Static relevance and causal usefulness are different quantities in agentic retrieval, and optimizing the first does not deliver the second.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。