arXiv:2504.07164cs.SEcs.CL2025-04被引 123

构建大规模可生成代码环境,提升开源编程模型性能至新高度

R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents

  • 用自动生成任务和回译技术构建8.7K个可执行训练环境
  • 结合两种验证方式,使开源模型在SWE-Bench上达到51%通过率
  • 适合研究开源编程代理、自动化测试与模型评估的开发者

提升开源模型在真实软件工程任务(如解决GitHub问题)上的表现面临两大挑战:一是可扩展地构建执行环境以训练模型,二是最优地扩展推理时计算资源。本文提出AgentGym,目前最大的程序化构建的可执行环境集合,包含超过8.7K个任务。其核心贡献包括:1)SYNGEN:一种合成数据构建方法,直接从提交记录中生成测试并回译,实现可扩展的环境构建,减少对人工编写问题或单元测试的依赖;使用该方法训练的32B模型在SWE-Bench Verified基准上达到pass@1 34.4%的性能。2)混合推理时扩展:深入分析基于执行和无执行的验证方式,发现二者具有互补性——前者区分度低,后者存在风格偏差。令人意外的是,单独使用任一方法均在42-43%饱和,而结合两者可显著提升性能。最终整体方法在SWE-Bench Verified上达到51%,刷新开源模型新纪录,首次在工具支持下达到与o1、o1-preview和sonnet-3.5-v2等闭源模型相当的水平。项目将开源环境、模型及代理轨迹。

原文摘要 · Abstract (English)

Improving open-source models on real-world SWE tasks (solving GITHUB issues) faces two key challenges: 1) scalable curation of execution environments to train these models, and, 2) optimal scaling of test-time compute. We introduce AgentGym, the largest procedurally-curated executable gym environment for training real-world SWE-agents, consisting of more than 8.7K tasks. AgentGym is powered by two main contributions: 1) SYNGEN: a synthetic data curation recipe that enables scalable curation of executable environments using test-generation and back-translation directly from commits, thereby reducing reliance on human-written issues or unit tests. We show that this enables more scalable training leading to pass@1 performance of 34.4% on SWE-Bench Verified benchmark with our 32B model. 2) Hybrid Test-time Scaling: we provide an in-depth analysis of two test-time scaling axes; execution-based and execution-free verifiers, demonstrating that they exhibit complementary strengths and limitations. Test-based verifiers suffer from low distinguishability, while execution-free verifiers are biased and often rely on stylistic features. Surprisingly, we find that while each approach individually saturates around 42-43%, significantly higher gains can be obtained by leveraging their complementary strengths. Overall, our approach achieves 51% on the SWE-Bench Verified benchmark, reflecting a new state-of-the-art for open-weight SWE-agents and for the first time showing competitive performance with proprietary models such as o1, o1-preview and sonnet-3.5-v2 (with tools). We will open-source our environments, models, and agent trajectories.

编程代理代码生成测试验证开源模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。