用学术研讨会生成研究任务,测试大模型真实科研能力
DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks
- 从真实研讨会中提取研究灵感,自动生成高质量科研任务
- 构建超1万条跨12个学科的任务库,挑战当前顶尖研究代理
- 适合评估大模型科研推理能力,尤其关注真实研究场景
深度研究智能体因其能协调多阶段研究流程(包括文献综述、方法设计和实证验证)而受到越来越多关注。然而,由于难以获取真正激发研究人员兴趣的前沿问题,对其科研能力的评估仍具挑战。为填补这一空白,我们提出了DeepResearch Arena,一个基于学术研讨会的基准评测体系,该体系捕捉专家间的丰富讨论与互动,更贴近真实研究环境,并降低数据泄露风险。我们设计了多智能体分层任务生成系统(MAHTG),从研讨会转录文本中提取具有研究价值的启发点,并将其转化为高质量研究任务,确保任务生成过程可追溯且去噪。借助该系统,我们从超过200场学术研讨会中构建了包含超1万项高质量研究任务的数据库,覆盖文学、历史、科学等12个学科。大规模评估表明,DeepResearch Arena对当前最先进的研究代理构成显著挑战,各模型间存在明显性能差距。
原文摘要 · Abstract (English)
Deep research agents have attracted growing attention for their potential to orchestrate multi-stage research workflows, spanning literature synthesis, methodological design, and empirical verification. Despite these strides, evaluating their research capability faithfully is rather challenging due to the difficulty of collecting frontier research questions that genuinely capture researchers' attention and intellectual curiosity. To address this gap, we introduce DeepResearch Arena, a benchmark grounded in academic seminars that capture rich expert discourse and interaction, better reflecting real-world research environments and reducing the risk of data leakage. To automatically construct DeepResearch Arena, we propose a Multi-Agent Hierarchical Task Generation (MAHTG) system that extracts research-worthy inspirations from seminar transcripts. The MAHTG system further translates research-worthy inspirations into high-quality research tasks, ensuring the traceability of research task formulation while filtering noise. With the MAHTG system, we curate DeepResearch Arena with over 10,000 high-quality research tasks from over 200 academic seminars, spanning 12 disciplines, such as literature, history, and science. Our extensive evaluation shows that DeepResearch Arena presents substantial challenges for current state-of-the-art agents, with clear performance gaps observed across different models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。