arXiv:2609.09133cs.AIcs.CL2026-09

让代码助手先写测试,再根据测试反馈改代码,提升修复准确率。

ExecCritic: Learn to Test, Test to Improve for Coding Agents

论文配图:ExecCritic: Learn to Test, Test to Improve for Coding Agents
图 1 · 摘自论文原文
  • 分角色训练:测试生成与代码修复分离,避免错误协同。
  • 使用高质量测试可使修复成功率从61.2%提升至65.3%。
  • 无需更强模型或人工反馈,仅靠角色训练即可显著提效。

执行反馈可引导代码代理完成仓库修复,但前提是测试能准确捕捉问题需求。代理自动生成的测试可能包含不完整或错误的行为目标;当同一路径同时生成补丁和测试时,错误可能一致,产生虚假信心。我们提出ExecCritic,结合测试-验证-修正框架与角色专属强化学习策略,训练其中的代理。该框架将测试构建与源码修复分离:测试代理独立生成项目原生测试,封闭式测试套件验证并冻结测试,修复代理基于其执行反馈修正代码而不修改测试。两个角色均以Qwen-3.5-35B-A3B为骨干,分别训练。在‘学写测试’阶段,测试代理学习生成能区分正确与错误补丁的行为有效测试;在‘以测促修’阶段,修复代理同时学习直接解决问题和反馈驱动的迭代优化。在SWE-bench Verified上,测试质量决定反馈是否有效:固定基础修复代理,基础测试代理使修复率从无测试基线的61.2%降至57.3%,而GPT-5.6-sol生成的测试则提升至65.3%。角色专属微调使Qwen测试代理的基线到黄金成功率从22.2%升至62.2%;两者联合后达到72.6%,比原始无测试基线高出11.4个百分点,且无需更强模型或评估时的真值反馈。代码公开于https://github.com/MSR-Orchard/execcritic。

原文摘要 · Abstract (English)

Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue. Agent-generated tests can encode incomplete or incorrect behavioral targets; when the same trajectory writes both the patch and the test, their errors can agree and create false confidence. We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it. The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests. Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately. In Learn to Test, the Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches. In Test to Improve, the Repair agent learns both direct task resolution and feedback-guided revision. On SWE-bench Verified, test quality determines whether feedback helps: holding the base Repair agent fixed, tests from the base Test agent reduce resolved rate from a no-test baseline of 61.2% to 57.3%, whereas tests from GPT-5.6-sol raise it to 65.3%. Role-specific post-training raises the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%; composing the two post-trained Qwen agents reaches 72.6%, an 11.4-point gain over the original no-test baseline without stronger-model or Oracle feedback at evaluation time. Code is publicly available at https://github.com/MSR-Orchard/execcritic.

代码生成强化学习测试驱动AI代理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。