构建首个长周期代理搜索评测基准，用智能裁判自动评估复杂答案。
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
- 设计树状评分框架，由任务专用裁判代理自动评估答案与引用。
- 覆盖130个真实长周期任务，耗时超1000小时人工构建。
- 首次系统评测十款前沿代理搜索系统，揭示性能差距与改进方向。
代理式搜索（如Deep Research系统）通过自主浏览网页、整合信息并返回有引用支持的全面回答，正重塑用户获取网络信息的方式。尽管能提升效率并减轻认知负担，其日益复杂的开放性任务已超出传统评估基准的覆盖范围——后者多假设短周期和静态答案。本文提出Mind2Web 2，一个包含130个高真实度、长周期任务的基准，需实时网页浏览与深度信息整合，构建耗时超过1000小时的人工标注。为应对动态复杂答案的评估难题，我们提出全新的Agent-as-a-Judge框架：基于树状评分结构，构建任务特定的裁判代理，自动评估答案正确性与来源溯源。我们对十款前沿代理搜索系统及人类表现进行综合评估，并开展详细错误分析，揭示未来发展方向。表现最佳的OpenAI Deep Research系统已达人类50%-70%的水平，且用时仅为一半，展现出巨大潜力。总体而言，Mind2Web 2为下一代代理搜索系统的开发与评测提供了严谨基础。
原文摘要 · Abstract (English)
Agentic search such as Deep Research systems-where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers-represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。