用细粒度验证提升论文代码复现准确率
Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification
- 基于论文指纹生成代码并迭代验证修正
- 在PaperBench上比基线高13.0%准确率
- 适合需要高精度复现的科研与工程人员
复现机器学习论文对科学进步至关重要,但对人工和自动化代理而言仍具挑战。现有基于智能体的方法常难以完整准确地复现数学公式和算法逻辑。已有研究显示,带有明确反馈的反思能提升智能体性能,但当前复现方法未能有效采用该策略。这一差距主要源于论文模式多样、方法模块复杂及配置多变。受人类使用系统检查表高效调试代码的启发,我们提出RePro框架——一种反射式论文到代码复现方法。该框架自动提取论文指纹,即一组精准且原子化的评估标准,作为高质量监督信号。首先基于提取信息生成代码,再通过指纹驱动的迭代验证与修正循环,系统检测差异并针对性修改,使生成代码与论文实现细节对齐。在PaperBench Code-Dev基准上的大量实验表明,RePro相较基线提升13.0%性能,尤其在复现复杂逻辑与数学规则方面表现显著。
原文摘要 · Abstract (English)
Reproducing machine learning papers is essential for scientific progress but remains challenging for both humans and automated agents. Existing agent-based methods often struggle to fully and accurately reproduce implementation details such as mathematical formulas and algorithmic logic. Previous studies show that reflection with explicit feedback improves agent performance. However, current paper reproduction methods fail to effectively adopt this strategy. This gap mainly arises from the diverse paper patterns, complex method modules, and varied configurations encountered in research papers. Motivated by how humans use systematic checklists to efficiently debug complex code, we propose \textbf{RePro}, a \textbf{Re}flective Paper-to-Code \textbf{Repro}duction framework that automatically extracts a paper's fingerprint, referring to a comprehensive set of accurate and atomic criteria serving as high-quality supervisory signals. The framework first generates code based on the extracted information, and then leverages the fingerprint within iterative verification and refinement loop. This approach systematically detects discrepancies and produces targeted revisions to align generated code with the paper's implementation details. Extensive experiments on the PaperBench Code-Dev benchmark have been conducted, RePro achieves 13.0\% performance gap over baselines, and it correctly revises complex logical and mathematical criteria in reflecting, on which the effectiveness is obvious.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。