AI可仅凭行为复现完整程序,无需源码
MirrorCode: AI can rebuild entire programs from behavior alone
- 用行为反推代码,让AI重实现整个软件项目
- 最强模型在25个任务中达56%正确率,可复现1.6万行生物信息工具
- 适合研究自主编程、软件工程自动化的学者和开发者
AI编程能力快速提升,但现有基准多限于短任务,单次演示难以系统比较。为此,我们提出MirrorCode,一个基于完整软件项目重实现的长周期编程基准。在该基准中,AI代理需在无源码情况下复现原程序功能,通过端到端测试(含保留测试集)验证。25个目标程序覆盖Unix工具、数据序列化、生物信息学、解释器、静态分析、密码学与压缩等领域。当前最强模型在基准上取得56%平均得分,例如可复现1.6万行的gotree生物信息工具包——这一任务对人类工程师可能需数周时间。完成大型任务需较大推理预算,如单次尝试耗时19天、花费2600美元。结果表明,具备精确需求描述时,AI已能完成长周期软件工程任务,预示其将对软件工程产生深远影响。
原文摘要 · Abstract (English)
AI models are rapidly improving at autonomous coding, as shown by benchmark progress and one-off demonstrations such as AI implementing a C compiler. However, existing coding benchmarks tend to focus on shorter tasks, and one-off demonstrations are hard to compare systematically because they often have some human guidance, and are not standardized or repeated across models. To address these challenges, we introduce MirrorCode, a long-horizon coding benchmark based on reimplementing entire software projects. In MirrorCode, AI agents must replicate the functionalities of an existing program, without access to its source code. AI solutions must match the original program's output exactly on end-to-end tests, including held-out tests. MirrorCode's 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression. Existing AI models can already reimplement complex software, with the strongest model scoring 56% across the benchmark. For example, AI can reimplement gotree, a 16,000-line bioinformatics toolkit - a task that we believe would take weeks for a human engineer. However, studying the frontier of performance requires a larger inference budget than typical benchmarks, for example, \$2,600 over 19 days for a single attempt on a large task. We show that AI agents can already complete long-horizon software engineering tasks, especially when requirements are precisely specified. More broadly, our work suggests AI will have transformative effects on software engineering, as autonomous agents continue to improve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。