构建500个真实研究任务基准,测试智能体的广度与深度检索能力。
WANDR: A Benchmark for Wide and Deep Research

- 用层级化任务结构模拟多轮网页搜索,要求覆盖实体、关系与证据链
- 最高仅达软F1 0.363,硬F1 0.133,深度与规模越大性能越差
- 适合评估智能研究系统在尽职调查、文献综述等复杂场景表现
WANDR(Wide ANd Deep Research)是一个包含500个真实且具挑战性的数据收集任务的基准,每个任务要求系统发现满足特定条件的大量实体(广度),通过多轮协同网络搜索深入探究每个实体(深度),并返回可独立验证的记录,附带来源和原文摘录。任务以资格键层级结构表示,明确各层级的实体、关系、证据及数量要求;例如包含n家公司、每家m名员工、每人k份来源时,需生成n×m×k条记录。该结构支持市场测绘、尽职调查、文献综述、产品对比与人才寻源等多种工作流,目标记录数从数十到数千不等。WANDR采用任务专属裁判机制,重新抓取引用页面并逐条验证记录与证据的一致性,可评估当前及动态变化的事实。记录判断结果聚合为软/硬精度、召回率与F1分数,区分事实质量、覆盖率与层级完整性。任务源自去标识化的产品使用日志,经半自动化流程生成,并通过自动化校验、实证审计与必要的人工审查确保质量。我们评估了六款生产级研究系统,发现基准尚未饱和:高投入下最强系统仅达软F1 0.363,硬F1 0.133。性能随目标量级与层级深度增加而下降,主要瓶颈仍为发现不全、信息补充缺失与证据链构建不完整。基准与评估工具开源于https://github.com/perplexityai/wandr。
原文摘要 · Abstract (English)
WANDR (Wide ANd Deep Research) is a benchmark of 500 realistic, challenging data-collection tasks for research agents. Each task requires a system to discover a large set of entities that satisfy specified criteria (breadth), investigate each entity through multiple coordinated web searches (depth), and return independently verifiable records with supporting sources and excerpts. Tasks are represented as qualification key hierarchies that specify the entities, relationships, evidence, and required count at each level; a hierarchy with n companies, m employees per company, and k sources per employee requires n x m x k records. This structure supports diverse workflows such as market mapping, due diligence, literature review, product comparison, and talent sourcing, with targets ranging from dozens to thousands of records. WANDR replaces static gold answer sets with task-specific judges that refetch cited pages and verify each record against its evidence, allowing evaluation of current and changing facts. Record verdicts are aggregated into soft and hard precision, recall, and F1 scores that distinguish factual quality, coverage, and hierarchical completeness. The tasks are derived from de-identified product-usage logs and produced through a semi-automated pipeline with automated checks, empirical audits, and human review where needed. We evaluate six production research systems and find that the benchmark is far from saturated: at high effort, the strongest system reaches only 0.363 soft F1 and 0.133 hard F1. Performance degrades as target volume and hierarchy depth increase, with incomplete discovery, missing enrichment, and incomplete evidence construction remaining major bottlenecks. The benchmark and evaluation harness are available at https://github.com/perplexityai/wandr.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。