arXiv:2603.04751cs.AI2026-03被引 1

用虚拟世界评估搜索智能体,解决真实数据难、过时快、结果不可靠的问题

Evaluating the Search Agent in a Parallel World

  • 构建平行世界模拟未来场景,生成不可篡改的原子事实与答案
  • 实测显示智能体在陌生环境收集证据能力弱,判断何时停止搜索也不准
  • 适合研究搜索增强型大模型性能瓶颈的开发者和评测人员

将网络搜索工具融入大语言模型显著拓展了其处理开放世界、实时及长尾问题的能力。然而,评估此类搜索智能体面临严峻挑战:首先,构建高质量深度搜索基准成本过高,而未经验证的合成数据常含不可靠来源;其次,静态基准随互联网信息演变而失效,复杂查询因流行度上升变为简单检索,真相亦因时间推移过时;第三,归因模糊使表现难以区分是记忆还是实际搜索推理所致;第四,依赖特定商业搜索引擎导致可复现性差。为此,我们提出全新框架Mind-ParaWorld(MPW),在平行世界中评估搜索智能体。MPW采样真实实体名生成超出模型知识截止日期的未来场景与问题,通过平行世界法则模型构建一组不可分割的原子事实与唯一真值。评估时,智能体与基于这些原子事实动态生成搜索结果页(SERPs)的平行世界引擎交互,而非真实检索。我们发布MPW-Bench,涵盖19个领域共1,608个实例。三组实验表明,尽管智能体在已有信息下具备强证据整合能力,但其表现受限于陌生环境中的证据获取与覆盖不足,以及对证据充分性的误判和何时停止搜索的决策瓶颈。

原文摘要 · Abstract (English)

Integrating web search tools has significantly extended the capability of LLMs to address open-world, real-time, and long-tail problems. However, evaluating these Search Agents presents formidable challenges. First, constructing high-quality deep search benchmarks is prohibitively expensive, while unverified synthetic data often suffers from unreliable sources. Second, static benchmarks face dynamic obsolescence: as internet information evolves, complex queries requiring deep research often degrade into simple retrieval tasks due to increased popularity, and ground truths become outdated due to temporal shifts. Third, attribution ambiguity confounds evaluation, as an agent's performance is often dominated by its parametric memory rather than its actual search and reasoning capabilities. Finally, reliance on specific commercial search engines introduces variability that hampers reproducibility. To address these issues, we propose a novel framework, Mind-ParaWorld, for evaluating Search Agents in a Parallel World. Specifically, MPW samples real-world entity names to synthesize future scenarios and questions situated beyond the model's knowledge cutoff. A ParaWorld Law Model then constructs a set of indivisible Atomic Facts and a unique ground-truth for each question. During evaluation, instead of retrieving real-world results, the agent interacts with a ParaWorld Engine Model that dynamically generates SERPs grounded in these inviolable Atomic Facts. We release MPW-Bench, an interactive benchmark spanning 19 domains with 1,608 instances. Experiments across three evaluation settings show that, while search agents are strong at evidence synthesis given complete information, their performance is limited not only by evidence collection and coverage in unfamiliar search environments, but also by unreliable evidence sufficiency judgment and when-to-stop decisions-bottlenecks.

搜索智能体评测基准虚拟世界大模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。