测试编码代理能否自动完成整个代码库的迁移,发现成功率不足6%。
SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

- 设计三阶段评估流程,验证迁移是否真正发生
- 仅5.4%的尝试通过全部测试,最佳模型得分47分
- 迁移完整性和行为正确性是两个独立能力
现代软件系统历经数十年开发积累了大量技术债,迁移成本高昂且主要依赖人工。随着编码代理在修复漏洞方面能力增强,它们能否自主完成此类迁移?现有基准无法回答此问题,因仅评估行为正确性,导致代理可通过复制原代码让测试通过(称作‘盲点’)。为此,我们提出SWE Refactor Bench,包含20个完整仓库迁移任务,覆盖4类技术债。采用三阶段评估协议:(1) 迁移审计,确认迁移是否发生;(2) 行为测试,使用固定测试集衡量正确性;(3) 代理验证,由6个独立编码代理生成针对性测试以捕捉隐藏行为差异。在8个前沿模型、26种配置下共520次运行中,仅28次(5.4%)通过所有阶段,20项任务中有13项无有效解,最优模型claude-opus-5得分为47.0/100。迁移完整性和行为正确性相互独立:少数运行跳过迁移但被审计阶段拦截;多数尝试迁移但破坏行为,被测试阶段拦截。在通过迁移审计的340次运行中,58%达到99%的固定检查,但仅26%达到100%。不同迁移类型表现差异显著:构建工具链重写得分31.4,语言重写仅5.6。这些发现表明SWE Refactor Bench是评估编码代理实现可靠全库迁移能力的严格基准。
原文摘要 · Abstract (English)
Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs ($5.4\%$) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores $47.0/100$. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, $58\%$ reach $99\%$ of the fixed checks, yet only $26\%$ reach $100\%$. Agent capability differs across migration categories: agents score $31.4$ on build toolchain rewrites but only $5.6$ on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。