评测大模型代理复现自然语言模型论文代码的能力
LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
- 构建28个来自顶会论文的代码复现任务，覆盖9类基础研究
- 最先进模型在单元测试中准确率仍不足，暴露科学推理短板
- 适合评估大模型科研自动化能力，尤其关注代码生成与理解
大语言模型代理在推动科学发现方面展现出巨大潜力，但在复现研究论文代码这一基础而关键的任务上，尤其是在自然语言处理领域，其能力仍缺乏深入探索。该任务涉及抽象概念整合与多文件依赖代码库理解等复杂推理挑战。为此，我们提出LMR-BENCH，一个系统性评估大模型代理从语言建模研究中复现代码能力的基准。它包含28个代码复现任务，源自过去五年发表于顶级NLP会议的23篇论文，覆盖九个基础类别。评估时提供研究论文、含一个或多个掩码函数的代码仓库及实现指令。我们在标准提示和大模型代理设置下，使用最先进的大模型进行实验，通过单元测试准确率和基于大模型的代码正确性评估进行衡量。结果表明，即使是最先进的模型，在科学推理与代码合成方面仍存在持续局限，凸显了大模型代理在自主复现科研成果方面的显著差距。
原文摘要 · Abstract (English)
Large language model (LLM) agents have demonstrated remarkable potential in advancing scientific discovery. However, their capability in the fundamental yet crucial task of reproducing code from research papers, especially in the NLP domain, remains underexplored. This task includes unique complex reasoning challenges in the intellectual synthesis of abstract concepts and the comprehension of code repositories with interdependent files. Motivated by this gap, we present LMR-BENCH, a benchmark designed to systematically evaluate the capability of LLM agents on code reproduction from Language Modeling Research. It consists of 28 code reproduction tasks derived from 23 research papers published in top-tier NLP venues over the past five years, spanning nine fundamental categories. Models are provided with a research paper, a code repository containing one or more masked functions, and instructions for implementing these functions. We conduct extensive experiments in standard prompting and LLM agent settings with state-of-the-art LLMs, evaluating the accuracy of unit tests and performing LLM-based evaluation of code correctness. Experimental results reveal that even the most advanced models still exhibit persistent limitations in scientific reasoning and code synthesis, highlighting critical gaps in LLM agents' ability to autonomously reproduce scientific research
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。