测试大模型复现最新机器学习论文代码的能力，发现最佳模型成功率不足四成。
ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code
- 构建212个前沿论文代码挑战，评估模型复现能力
- 顶级模型成功率仅37.3%，多数未达40%
- 揭示错误模式与数据污染问题，助力改进研究代码生成
大型语言模型（LLMs）在推动机器学习研究方面展现出潜力，但其将未经预训练的最新研究成果转化为可执行代码的能力尚不明确。我们提出ResearchCodeBench，一个包含212个编程挑战的基准，用于评估模型从2024-2025年顶会论文中复现前沿机器学习方法的能力。我们测试了30多个专有及开源模型，发现即使表现最好的模型也仅能正确实现不足40%的代码。Gemini-2.5-Pro-Preview以37.3%的成功率领先，O3（High）和O4-mini（High）分别达到32.3%和30.8%。研究揭示了性能差异、数据污染及常见错误模式。通过提供一个严谨且社区驱动的评估平台，ResearchCodeBench有助于持续理解并推动研究代码生成中的大模型创新。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown promise in transforming machine learning research, yet their capability to faithfully implement novel ideas from recent research papers-ideas unseen during pretraining-remains unclear. We introduce ResearchCodeBench, a benchmark of 212 coding challenges that evaluates LLMs' ability to translate cutting-edge ML contributions from top 2024-2025 research papers into executable code. We assessed 30+ proprietary and open-source LLMs, finding that even the best models correctly implement less than 40% of the code. We find Gemini-2.5-Pro-Preview to perform best at 37.3% success rate, with O3 (High) and O4-mini (High) following behind at 32.3% and 30.8% respectively. We present empirical findings on performance comparison, contamination, and error patterns. By providing a rigorous and community-driven evaluation platform, ResearchCodeBench enables continuous understanding and advancement of LLM-driven innovation in research code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。