arXiv:2606.18237cs.CLcs.AI2026-06

用GitHub问题自动评估论文复现难题,提升研究可重复性检测效率。

ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

论文配图:ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
图 1 · 摘自论文原文
  • 利用真实GitHub问题作为监督信号,自动识别复现障碍。
  • 最佳模型在90%论文中发现至少一个相关复现问题。
  • 适合关注科研可复现性、评测AI助手能力的研究者使用。

论文复现是科学进步的核心。现有研究虽提出评估大模型代理在复现任务中作用的基准,但因依赖大量人工数据整理与评估,难以扩展。本文提出ReproRepo,一种可扩展的复现评估框架,利用开发者在GitHub仓库中提出的实际问题作为自然监督信号,识别真实的复现障碍。我们在1,149篇顶级会议的机器学习论文上实例化该框架,并评估了四种前沿模型代理配置。结果显示,即使不运行代码,大模型代理也能从论文-代码配对中识别出大量现实复现问题:最佳代理Codex结合GPT-5.5,在约90%的论文中发现了至少一个语义相关的用户报告问题。进一步分析表明,代理在识别明显失败和正确语义区域方面表现优异,但在精确定位上仍不足。ReproRepo可作为未来评估大模型在真实复现审计中表现的可复用、可扩展框架。代码已开源:https://github.com/LithiumDA/ReproRepo。

原文摘要 · Abstract (English)

Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist with reproducibility, but they are difficult to scale due to their reliance on substantial manual effort for data curation and evaluation. We introduce ReproRepo, a scalable framework for reproducibility evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers. We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations. Our results show that LLM agents, even without executing code, can identify many real-world reproducibility problems from paper-repository pairs: the best agent in our study, namely Codex with GPT-5.5, surfaces at least one semantically related human-reported blocker for $\sim$90% of papers in the study. Further analysis shows that agents are particularly effective for surfacing visible failures and identifying the right semantic region, but may still be insufficient in exact localization. ReproRepo can serve as a reusable, scalable framework for future evaluations of LLM agents on real-world reproducibility auditing. Our code is released at https://github.com/LithiumDA/ReproRepo.

可复现性大模型评估GitHub挖掘

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。