用AI自动复现科研论文,让结果可信度一目了然
VERITAS: Towards a General-Purpose Replication Tool for Scientific Research

- 构建通用命令行编码代理框架,自动解析论文并运行实验
- 在65篇跨学科论文上复现成功率达新高,优于两个强基线模型
- 适合研究人员、审稿人和期刊,提升科研可验证性
AI工具加速科学出版,但评审系统难以跟上,独立复现研究变得愈发困难且重要。手动复现耗时费力,现有方法多为封闭式基准测试,缺乏通用工具。本文提出VERITAS,一个面向任意领域的复现框架,基于命令行编码代理。给定论文或代码仓库,VERITAS提取论文主张,执行方法并实时修复问题,根据实验结果评估每项主张。输出包括加权复现得分、修复记录日志及修改后的代码库。在CORE-Bench与ReplicationBench上对65篇涵盖计算机科学、社会科学、医学与天体物理的论文进行评估,相较于两个同环境下的Claude Code基线,VERITAS在两项基准上所有指标均达到领先水平。
原文摘要 · Abstract (English)
AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important. As manual replication is slow and expensive, a growing line of work uses coding agents to automate parts of the process. Existing efforts are largely packaged as benchmarks with companion agents that only run inside the benchmark's own pipeline, and no general-purpose replication tool exists. We present VERITAS, a domain-agnostic replication framework built around CLI coding agents. Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs. The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched codebase. We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics. Against two strong Claude Code baselines on the same model and host environment, VERITAS achieves state-of-the-art performance and leads on every metric on both benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。