用AI自动评估论文可复现性，提升学术评审效率。
ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review
- 将可复现性评估转化为结构化推理任务，构建实验流程图。
- 在213篇论文上达到60%以上准确率，优于现有方法。
- 适合科研管理者与期刊审稿人，助力大规模科学审查。
科学同行评审正面临现代研究规模与复杂性带来的可复现性评估挑战。评估可复现性需重构实验依赖、方法选择、数据流和结果生成过程，远超人工评审能力。本文提出代理式可复现性评估（ARA），将可复现性评估形式化为对科学文献的结构化推理任务。给定一篇论文，ARA提取包含源、方法、实验与输出的有向工作流图，并基于结构与内容得分评估其可重建性。在213篇ReScience C文章（当前最大跨领域人类验证计算可复现性基准）上的实验表明，ARA在不同LLM、模型温度及科学领域间均具泛化能力。在三个基准上平均准确率达~61%，在ReproBench（60.71% vs. 36.84%）和GoldStandardDB（61.68% vs. 43.56%）上达到最高记录，凸显其规模化辅助人工评审的潜力。代码与数据已公开：https://github.com/AndresLaverdeMarin/agentic_reproducibility_assessment。
原文摘要 · Abstract (English)
Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibility requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, which often exceeds what human reviewers can provide. Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a paper, ARA extracts a directed workflow graph linking sources, methods, experiments, and outputs, then evaluates its reconstructability using structural and content-based scores for reproducibility assessments. Experiments on 213 ReScience C articles - the largest cross-domain benchmark of human-validated computational reproducibility studies considered to date - demonstrate ARA's generalizability and consistent workflow reconstruction and assessment across LLMs, model temperatures, and scientific domains. ARA achieves ~61% accuracy on three benchmarks, and the highest accuracy reported on ReproBench (60.71% vs. 36.84%) and GoldStandardDB (61.68% vs. 43.56%), highlighting its potential to complement human review at scale and enabling next-generation peer review. Code and Data available: https://github.com/AndresLaverdeMarin/agentic_reproducibility_assessment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。