测试AI能否复现顶会论文，发现当前最强模型仅达21%成功率。
PaperBench: Evaluating AI's Ability to Replicate AI Research
- 构建8316个可评分任务，按论文贡献拆解复现步骤
- 顶尖AI模型复现成功率仅21%，仍不及人类专家
- 引入作者协作设计评分标准，支持自动评估
我们提出PaperBench，一个评估AI代理复现前沿人工智能研究能力的基准。代理需从零开始复现20篇ICML 2024亮点与口头报告论文，涵盖理解论文贡献、搭建代码库和完成实验。为实现客观评估，我们开发了分层评分体系，将每项复现任务分解为更小子任务，并设定明确评分标准。总计包含8,316个独立可评分任务。评分标准由原论文作者共同制定，确保准确性和真实性。为支持可扩展评估，我们还开发基于LLM的评判器，自动依据评分标准打分，并通过另建基准评估其表现。在纸牌上对多个前沿模型进行评估，结果显示表现最佳的代理（Claude 3.5 Sonnet新版本 + 开源框架）平均复现得分为21.0%。我们还招募顶级机器学习博士尝试部分任务，结果表明当前模型尚未超越人类基准。代码已开源（https://github.com/openai/preparedness），以促进对AI工程能力的进一步研究。
原文摘要 · Abstract (English)
We introduce PaperBench, a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch, including understanding paper contributions, developing a codebase, and successfully executing experiments. For objective evaluation, we develop rubrics that hierarchically decompose each replication task into smaller sub-tasks with clear grading criteria. In total, PaperBench contains 8,316 individually gradable tasks. Rubrics are co-developed with the author(s) of each ICML paper for accuracy and realism. To enable scalable evaluation, we also develop an LLM-based judge to automatically grade replication attempts against rubrics, and assess our judge's performance by creating a separate benchmark for judges. We evaluate several frontier models on PaperBench, finding that the best-performing tested agent, Claude 3.5 Sonnet (New) with open-source scaffolding, achieves an average replication score of 21.0%. Finally, we recruit top ML PhDs to attempt a subset of PaperBench, finding that models do not yet outperform the human baseline. We open-source our code (https://github.com/openai/preparedness) to facilitate future research in understanding the AI engineering capabilities of AI agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。