arXiv:2605.28721cs.AI2026-05被引 2

研究发现大模型搜索代理其实靠记忆验证,而非真正搜索。

LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?

论文配图:LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?
图 1 · 摘自论文原文
  • 用三个诊断方法分析搜索行为,发现模型依赖内部知识
  • 44.5%问题无需工具即可回答,超半数查询源于内部假设
  • 新基准LiveBrowseComp考验实时搜索能力,适合评估真搜索性能

LLM-based搜索代理是真正搜索,还是仅用网络验证已有知识?我们在BrowseComp上通过三种诊断方法进行研究。结果揭示了内在知识依赖(IKD):即使有工具访问权限,代理仍主要依赖模型预训练时编码的内部知识,而非外部证据。代理在无工具情况下可回答高达44.5%的问题,超过一半的搜索查询源自内部生成的假设,而非检索到的线索;当支持性证据被移除后,其表现反而劣于封闭书本基线。这表明静态基准可能奖励记忆验证而非证据驱动发现,混淆了“已知”与“可查”。为此,我们提出LiveBrowseComp,一个深度搜索基准,包含335个由人类撰写的题目,答案依赖于基准构建前90天内发布的事实,数据源来自六个动态更新的渠道,并排除全球显著事件。在LiveBrowseComp上,所有评测代理的闭书准确率均低于2%,搜索增强得分相比BrowseComp下降25-40分,且以往模型排名不再可靠。LiveBrowseComp 已开放于 https://huggingface.co/datasets/Forival/LiveBrowseComp。

原文摘要 · Abstract (English)

Are LLM-based search agents genuinely searching, or using the web to verify what they already know? We study this question on BrowseComp with three diagnostics. Our analysis reveals Intrinsic Knowledge Dependence (IKD): even with tool access, agents often rely on intrinsic knowledge -- information encoded in the model before retrieval -- rather than on external evidence. Agents answer up to 44.5% of BrowseComp questions without tools, generate more than half of their search queries from internally produced hypotheses rather than retrieved leads, and perform worse than closed-book baselines when answer-supporting evidence is removed. These results suggest that static search benchmarks can reward memory-backed verification rather than evidence-driven discovery, conflating what agents already know with what they can find. We then introduce LiveBrowseComp, a deep-search benchmark designed to evaluate agents beyond intrinsic coverage. It contains 335 human-authored questions whose answers depend on facts published within the 90 days preceding benchmark construction, drawn from six updated sources and filtered to exclude globally salient events. On LiveBrowseComp, all evaluated agents fall below 2% closed-book accuracy, search-augmented scores drop by 25-40 points relative to BrowseComp, and prior model rankings no longer reliably predict performance. LiveBrowseComp is available at https://huggingface.co/datasets/Forival/LiveBrowseComp.

搜索代理知识依赖动态评估基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。