评估进化计算论文可复现性,发现仅三成论文提供额外材料。
Assessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessment
- 设计检查清单,人工与LLM结合评估论文可复现性。
- 论文平均完整度0.62,36.9%提供补充材料。
- 自动化工具与人工评估一致性高,适合大规模监控。
可复现性是进化计算中的关键要求,其结果高度依赖于计算实验。实际中,可复现性取决于算法、实验协议和成果物的文档化与共享程度。尽管关注度上升,但该领域发表工作的实际可复现水平仍缺乏实证证据。本文研究了十年间遗传与进化计算会议中进化组合优化与元启发式方向论文的可复现实践。我们引入结构化可复现性检查清单,并通过系统性人工评估选定文献集。此外,提出基于LLM的自动化评估系统RECAP(REproducibility Checklist Automation Pipeline),从论文文本和代码库自动提取可复现信号。分析显示,论文平均完整性得分为0.62,36.90%提供了稿件之外的附加材料。结果表明自动化评估可行:RECAP与人类评估者达成较高一致性(Cohen's k = 0.67)。整体揭示了可复现报告中的持续缺口,并建议自动化工具可用于大规模、系统性的可复现性监控。
原文摘要 · Abstract (English)
Reproducibility is an important requirement in evolutionary computation, where results largely depend on computational experiments. In practice, reproducibility relies on how algorithms, experimental protocols, and artifacts are documented and shared. Despite growing awareness, there is still limited empirical evidence on the actual reproducibility levels of published work in the field. In this paper, we study the reproducibility practices in papers published in the Evolutionary Combinatorial Optimization and Metaheuristics track of the Genetic and Evolutionary Computation Conference over a ten-year period. We introduce a structured reproducibility checklist and apply it through a systematic manual assessment of the selected corpus. In addition, we propose RECAP (REproducibility Checklist Automation Pipeline), an LLM-based system that automatically evaluates reproducibility signals from paper text and associated code repositories. Our analysis shows that papers achieve an average completeness score of 0.62, and that 36.90% of them provide additional material beyond the manuscript itself. We demonstrate that automated assessment is feasible: RECAP achieves substantial agreement with human evaluators (Cohen's k of 0.67). Together, these results highlight persistent gaps in reproducibility reporting and suggest that automated tools can effectively support large-scale, systematic monitoring of reproducibility practices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。