AI能自己做科研实验吗？这个新基准测试发现目前还基本做不到。
EXP-Bench: Can AI Conduct AI Research Experiments?
- 构建了461个来自顶会论文的真实科研实验任务，评估AI完成全流程实验的能力。
- 主流AI代理在完整实验中成功率仅0.5%，各环节正确率最高达35%。
- 适合研究AI自动化、智能体系统与科研自主性的学者使用。
自动化人工智能研究具有加速科学进步的巨大潜力，但当前AI代理在复杂、端到端的实验中仍表现不佳。我们提出EXP-Bench，一个新型基准，用于系统评估AI代理在源自重要人工智能论文的完整研究实验中的表现。给定研究问题和不完整的起始代码，EXP-Bench要求AI代理制定假设、设计并实现实验流程、执行实验并分析结果。为生成高保真的复杂任务，我们设计了一个半自动化流水线，从论文及其开源代码中提取并结构化关键实验细节。通过该流水线，EXP-Bench从51篇顶级人工智能论文中整理出461个研究任务。对OpenHands和IterativeAgent等主流LLM驱动代理的评估显示，其在单个实验环节（如设计或实现）的正确率偶尔可达20%-35%，但在完整可执行实验中的成功率为0.5%。通过识别这些瓶颈并提供真实、分步的实验流程，EXP-Bench为未来AI代理提升科研实验能力提供了关键工具。项目已开源：https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench。
原文摘要 · Abstract (English)
Automating AI research holds immense potential for accelerating scientific progress, yet current AI agents struggle with the complexities of rigorous, end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed to systematically evaluate AI agents on complete research experiments sourced from influential AI publications. Given a research question and incomplete starter code, EXP-Bench challenges AI agents to formulate hypotheses, design and implement experimental procedures, execute them, and analyze results. To enable the creation of such intricate and authentic tasks with high-fidelity, we design a semi-autonomous pipeline to extract and structure crucial experimental details from these research papers and their associated open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks from 51 top-tier AI research papers. Evaluations of leading LLM-based agents, such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial capabilities: while scores on individual experimental aspects such as design or implementation correctness occasionally reach 20-35%, the success rate for complete, executable experiments was a mere 0.5%. By identifying these bottlenecks and providing realistic step-by-step experiment procedures, EXP-Bench serves as a vital tool for future AI agents to improve their ability to conduct AI research experiments. EXP-Bench is open-sourced at https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。