测试AI如何从部分代码逐步复现论文实验，揭示真实科研挑战。
From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking
- 用逐步掩码代码的方式，从复现到从零重写，评估AI实验能力。
- 随着缺失函数增多，成功率急剧下降，5次尝试才成功一次。
- 适合研究长程代码生成与自主实验的学者，推动智能科研发展。
自主代码生成的进展激发了对能加速科学发现的AI代理的期待。然而，当前缺乏基准来评估这些代理在不同代码起点下实现科学想法的能力，即从复现（运行已有代码）到从头复制（完全重构并运行代码）之间的连续过程。我们提出AutoExperiment，一个评估AI代理基于论文自然语言描述实现并运行机器学习实验能力的基准。每个任务中，代理接收一篇论文、一个关键函数被掩码的代码库，以及运行实验的指令。目标是生成缺失代码，在沙盒环境中执行实验并复现结果。该基准通过改变缺失函数数 $n$ 来调节难度，覆盖从部分复现到完全复制的全过程。我们评估了当前最先进的代理，发现性能随 $n$ 增加而快速下降。能够动态交互环境（如调试代码）的代理优于固定“无代理”框架，且单次尝试成功率（Pass@1）与五次尝试成功率（Pass@5）差距显著，表明需引入验证器机制。研究揭示了长程代码生成、上下文检索和自主实验执行中的关键挑战，确立AutoExperiment为新一代智能科研实验评估基准。数据与代码已开源：https://github.com/j1mk1m/AutoExperiment。
原文摘要 · Abstract (English)
Recent progress in autonomous code generation has fueled excitement around AI agents capable of accelerating scientific discovery by running experiments. However, there is currently no benchmark that evaluates whether such agents can implement scientific ideas when given varied amounts of code as a starting point, interpolating between reproduction (running code) and from-scratch replication (fully re-implementing and running code). We introduce AutoExperiment, a benchmark that evaluates AI agents' ability to implement and run machine learning experiments based on natural language descriptions in research papers. In each task, agents are given a research paper, a codebase with key functions masked out, and a command to run the experiment. The goal is to generate the missing code, execute the experiment in a sandboxed environment, and reproduce the results. AutoExperiment scales in difficulty by varying the number of missing functions $n$, ranging from partial reproduction to full replication. We evaluate state-of-the-art agents and find that performance degrades rapidly as $n$ increases. Agents that can dynamically interact with the environment (e.g. to debug their code) can outperform agents in fixed "agentless" harnesses, and there exists a significant gap between single-shot and multi-trial success rates (Pass@1 vs. Pass@5), motivating verifier approaches to our benchmark. Our findings highlight critical challenges in long-horizon code generation, context retrieval, and autonomous experiment execution, establishing AutoExperiment as a new benchmark for evaluating progress in AI-driven scientific experimentation. Our data and code are open-sourced at https://github.com/j1mk1m/AutoExperiment .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。