用多智能体自动评估社科论文可复现性，效果显著优于旧方法。
PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers
- 分两阶段设计：先执行代码复现，再评估结果真实性。
- 在基准测试中比最强基线提升21.9%的评分一致性准确率。
- 适合研究可复现性、自动化评估系统或社科数据验证的学者。
计算可复现性对社会科学的科学发现可信度至关重要，因其成果常影响现实决策。人工评估耗时费力，因复现作者发布的代码与数据极具挑战。大模型的进展催生了基于智能体的自动化评估方法，但现有方案常受限于上下文容量、任务工具不足和结果捕获不全。为此，我们提出PaperRepro，一种新型两阶段多智能体框架，将执行与评估分离。执行阶段，智能体运行复现包并修改代码，显式记录复现结果作为可验证产物；评估阶段，智能体基于明确证据判断可复现性。PaperRepro为各智能体分配专属职责，配备任务特定工具与专家提示，缓解上下文与工具瓶颈，并最大化大模型编码能力以实现更完整的结果捕获。在REPRO-Bench（一个社会科学研究可复现性评估基准）上，PaperRepro取得最佳综合性能，较最强基线相对提升21.9%的评分一致性准确率。我们进一步优化该基准，推出按执行难度分层的REPRO-Bench-S，支持更精准评估自动化可复现性系统。代码与数据已公开。
原文摘要 · Abstract (English)
Computational reproducibility is essential for the credibility of scientific findings, particularly in the social sciences, where findings often inform real-world decisions. Manual reproducibility assessment is costly and time-consuming, as it is nontrivial to reproduce the reported findings using the authors' released code and data. Recent advances in large models (LMs) have inspired agent-based approaches for automated reproducibility assessment. However, existing approaches often struggle due to limited context capacity, inadequate task-specific tooling, and insufficient result capture. To address these, we propose PaperRepro, a novel two-stage, multi-agent approach that separates execution from evaluation. In the execution stage, agents execute the reproduction package and edit the code to capture reproduced results as explicit artifacts. In the evaluation stage, agents evaluate reproducibility using explicit evidence. PaperRepro assigns distinct responsibilities to agents and equips them with task-specific tools and expert prompts, mitigating context and tooling limitations. It further maximizes the LM's coding capability to enable more complete result capture for evaluation. On REPRO-Bench, a social science reproducibility assessment benchmark, PaperRepro achieves the best overall performance, with a 21.9% relative improvement in score-agreement accuracy over the strongest prior baseline. We further refine the benchmark and introduce REPRO-Bench-S, a benchmark stratified by execution difficulty for more diagnostic evaluation of automated reproducibility assessment systems. Our code and data are publicly available
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。