评测智能体同时查资料算数据的能力,更贴近真实研究场景。
DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?
- 从知识图谱反推问题,生成需查资料+算数的合成题目。
- 顶尖模型仅20%答对,凸显当前智能体综合能力短板。
- 覆盖生化、金融等5个领域,适合评估复杂推理系统。
深度研究代理日益结合网页浏览与多步计算,但现有基准测试将这些能力分开评估,导致无法真实反映性能。我们提出DRBENCHER,一种从知识图谱链反向生成的问题合成器,不依赖初始文本或来源信息。该方法满足四个标准:可验证性(答案通过参数化代码在知识图谱值上执行得出)、复杂性(多跳实体识别、属性检索和领域特定计算)、难度(两阶段验证级联剔除可被生成模型解决的问题)和多样性(采用贪心最大最小嵌入过滤以最大化覆盖范围)。该标准通过统一的答案优先流程在五个领域(生物化学、金融、地球物理、安全、历史)实现。人工评估显示76%的问题有效,35%错误源于过时的知识图谱条目,揭示了基于动态数据推理系统的固有局限。自动评估显示最强前沿模型仅达20%答案准确率。相比手动构建的基准(BrowseComp+, MATH-500, GPQA),DRBENCHER具有最高的语义多样性。我们已公开DrBencher管道及经人工验证的基准数据集,地址为https://github.com/IBM/DrBencher。
原文摘要 · Abstract (English)
Deep research agents increasingly interleave web browsing with multi-step computation, yet existing benchmarks evaluate these capabilities in isolation, creating a blind spot in assessing real-world performance. We introduce DRBENCHER, a synthetic benchmark generator for questions that require both browsing and computation, synthesized answer-first from knowledge-graph chains with no seed passages or provenance text. It enforces four criteria: verifiability (gold answers are computed by executing parameterized code over knowledge-graph values), complexity (multi-hop entity identification, property retrieval, and domain-specific computation), difficulty (a two-stage verification cascade filters out questions solvable by the generating model), and diversity (a greedy max-min embedding filter maximizes coverage). These criteria are realized via a unified answer-first pipeline spanning five domains: biochemistry, financial, geophysical, security, and history. Human evaluation shows 76% validity, with 35% of errors due to outdated knowledge-graph entries, highlighting an inherent limitation of systems that reason over evolving data. Automatic evaluation shows that the strongest frontier model achieves only 20% answer accuracy. Compared to manually constructed benchmarks (BrowseComp+, MATH-500, GPQA), DRBENCHER achieves the highest semantic diversity. We release DrBencher pipeline and the human-verified benchmark data at https://github.com/IBM/DrBencher.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。