arXiv:2601.10504cs.CL2026-01ACL被引 3

用动态网页信息自动评估大模型研究能力,更真实可靠。

DR-Arena: an Automated Evaluation Framework for Deep Research Agents

  • 构建实时信息树,让评测紧跟网络最新趋势。
  • 通过自动化任务测试深度推理与广度覆盖能力,性能相关性达0.94。
  • 适合需要高效、低成本评估研究型AI的团队使用。

随着大语言模型日益作为能够自主调查与信息整合的深度研究代理(Deep Research Agents)运行,其任务表现的可靠评估已成为关键瓶颈。现有基准多依赖静态数据集,存在任务泛化性差、时间错配和数据污染等问题。为此,我们提出DR-Arena,一个全自动评估框架,通过动态调查将研究代理推向能力极限。DR-Arena基于实时网络趋势构建信息树,确保评估标准与现实世界状态同步,并采用自动化评审员生成结构化任务,测试两个正交能力:深度推理与广泛覆盖。框架还引入自适应演进循环,一种状态机控制器,根据实时表现动态提升任务复杂度,持续要求更深层次推理或更广范围聚合,直至出现决定性的能力边界。对六种先进研究代理的实验表明,DR-Arena与LMSYS Search Arena排行榜的斯皮尔曼相关系数达0.94,实现了无需人工干预的状态前沿对齐,验证了其作为昂贵人工评判的可靠替代方案。

原文摘要 · Abstract (English)

As Large Language Models (LLMs) increasingly operate as Deep Research (DR) Agents capable of autonomous investigation and information synthesis, reliable evaluation of their task performance has become a critical bottleneck. Current benchmarks predominantly rely on static datasets, which suffer from several limitations: limited task generality, temporal misalignment, and data contamination. To address these, we introduce DR-Arena, a fully automated evaluation framework that pushes DR agents to their capability limits through dynamic investigation. DR-Arena constructs real-time Information Trees from fresh web trends to ensure the evaluation rubric is synchronized with the live world state, and employs an automated Examiner to generate structured tasks testing two orthogonal capabilities: Deep reasoning and Wide coverage. DR-Arena further adopts Adaptive Evolvement Loop, a state-machine controller that dynamically escalates task complexity based on real-time performance, demanding deeper deduction or wider aggregation until a decisive capability boundary emerges. Experiments with six advanced DR agents demonstrate that DR-Arena achieves a Spearman correlation of 0.94 with the LMSYS Search Arena leaderboard. This represents the state-of-the-art alignment with human preferences without any manual efforts, validating DR-Arena as a reliable alternative for costly human adjudication.

大模型评估自动评测研究代理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。