arXiv:2602.08561cs.SEcs.CL2026-02被引 3

用AI自动修复社科计算研究中的复现失败,效果比传统方法好。

Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches

  • 用提示词和智能体两种方式自动诊断修复代码错误
  • 智能体系统复现成功率最高达96%,远超提示词方法
  • 适合希望提升论文可复现性的研究人员参考

复现计算研究常被认为只需重新运行原始代码即可,但实际中缺失依赖包、路径错误、版本冲突或逻辑不全等问题频繁导致分析失败,即使数据与代码已公开。本研究探讨大语言模型与AI智能体能否自动诊断并修复此类问题,从而提升结果的可复现性与可验证性。我们基于五个完全可复现的R语言社会科学研究构建了受控复现测试平台,并注入从简单到复杂的各类真实故障。在干净的Docker环境中测试两种自动化修复流程:第一种为提示词驱动,通过结构化提示反复调用语言模型;第二种为智能体系统,能自主检查文件、修改代码并重跑分析。提示词方法的复现成功率在31%-79%之间,受提示上下文和错误复杂度影响显著,复杂问题更需额外上下文。智能体方法表现更好,所有复杂度下的成功率均达69%-96%。结果表明,尤其是智能体系统,能显著降低人工干预,有效应对多种错误类型。不同于以往基准,本测试平台在受控故障模式下聚焦发表后的修复,实现对提示词与智能体方法的直接对比。

原文摘要 · Abstract (English)

Reproducing computational research is often assumed to be as simple as rerunning the original code with provided data. In practice, missing packages, fragile file paths, version conflicts, or incomplete logic frequently cause analyses to fail, even when materials are shared. This study investigates whether large language models and AI agents can automate the diagnosis and repair of such failures, making computational results easier to reproduce and verify. We evaluate this using a controlled reproducibility testbed built from five fully reproducible R-based social science studies. Realistic failures were injected, ranging from simple issues to complex missing logic, and two automated repair workflows were tested in clean Docker environments. The first workflow is prompt-based, repeatedly querying language models with structured prompts of varying context, while the second uses agent-based systems that inspect files, modify code, and rerun analyses autonomously. Across prompt-based runs, reproduction success ranged from 31-79 percent, with performance strongly influenced by prompt context and error complexity. Complex cases benefited most from additional context. Agent-based workflows performed substantially better, with success rates of 69-96 percent across all complexity levels. These results suggest that automated workflows, especially agent-based systems, can significantly reduce manual effort and improve reproduction success across diverse error types. Unlike prior benchmarks, our testbed isolates post-publication repair under controlled failure modes, allowing direct comparison of prompt-based and agent-based approaches.

可复现性AI修复智能体社会科学

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。