测试AI编程代理能否在真实科研任务中实现创新发现
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
- 构建90个跨学科任务，从顶刊论文中提炼科学问题
- 最强模型仅在17.8%任务上超越现有最优解
- 代理主要靠转化问题而非真正发明新方法，适合研究者评估智能体能力
我们提出NatureBench，一个由顶刊《Nature》系列论文提炼出的90个跨学科任务基准，旨在评估AI编程代理是否能超越复现，实现在真实科研问题上的发现。NatureBench基于NatureGym自动化流水线，从原始论文生成标准化、容器化的任务环境，解决了以往研究中因环境碎片化导致可信度不足的问题。在禁用网络搜索的严格协议下，对十种前沿代理配置进行评估，结果显示最强模型在g>0.1标准下仅在17.8%的任务上超越现有最优（SOTA）。分析表明，代理成功主要依赖方法迁移，将科学任务转化为熟悉的监督预测问题，而非真正的科学发明。失败主因是方法选择错误和算力不足，而非任务理解偏差。我们开源了基准、NatureGym流水线及公开排行榜，支持维护方复现。
原文摘要 · Abstract (English)
We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。