新基准通过专家标准评估研究型AI报告质量,发现顶尖模型仅达标一半。
DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report
- 基于专家调查报告构建9430条细粒度评分项,覆盖信息、分析、呈现三维度。
- 132个跨领域任务中,最强模型平均满足率不足50%,暴露显著能力差距。
- 适合评估大模型研究能力或改进智能调研系统的研究者使用。
深度研究系统(DRS)旨在辅助用户网络搜索、信息整合并生成综合调查报告,但其严谨评估方法仍不充分。现有基准常陷入两种缺陷:部分无法有效检验系统分析证据与撰写连贯报告的能力;另一些则依赖过于粗略或由大模型直接定义的评价标准,导致评分偏离人类专家判断且难以验证。为此,我们提出Deep Research Bench II,一个用于评估DRS生成报告的新基准。该基准包含22个领域的132个有据研究任务;每项任务要求生成长篇研究报告,并由总计9430条细粒度二元评分项进行评估,涵盖信息召回、分析深度和报告呈现三个维度。所有评分项均源自精心挑选的专家撰写的调查文章,通过结合自动提取与超过400小时人工专家评审的四阶段人机协作流程构建,确保标准原子化、可验证且贴近人类专家判断。我们在Deep Research Bench II上评估多个先进深度研究系统,发现即使最强模型也仅满足少于50%的评分项,揭示当前DRS与人类专家之间存在显著差距。
原文摘要 · Abstract (English)
Deep Research Systems (DRS) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. However, how to rigorously evaluate these systems remains under-explored. Existing deep-research benchmarks often fall into two failure modes. Some do not adequately test a system's ability to analyze evidence and write coherent reports. Others rely on evaluation criteria that are either overly coarse or directly defined by LLMs (or both), leading to scores that can be biased relative to human experts and are hard to verify or interpret. To address these issues, we introduce Deep Research Bench II, a new benchmark for evaluating DRS-generated reports. It contains 132 grounded research tasks across 22 domains; for each task, a system must produce a long-form research report that is evaluated by a set of 9430 fine-grained binary rubrics in total, covering three dimensions: information recall, analysis, and presentation. All rubrics are derived from carefully selected expert-written investigative articles and are constructed through a four-stage LLM+human pipeline that combines automatic extraction with over 400 human-hours of expert review, ensuring that the criteria are atomic, verifiable, and aligned with human expert judgment. We evaluate several state-of-the-art deep-research systems on Deep Research Bench II and find that even the strongest models satisfy fewer than 50% of the rubrics, revealing a substantial gap between current DRSs and human experts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。