让代码生成模型学会根据测试反馈逐步修复错误,提升多轮纠错能力。
TaPR: Test-Aware Policy Refinement for Feedback-Conditioned Code Generation
- 用每轮测试通过率构建密集奖励,精准反映修复过程
- 在219个任务上将三轮成功率提升2.44个百分点
- 特别适合需要持续优化的代码生成与调试场景
多轮代码生成代理依赖执行反馈来修复错误程序,但传统强化学习主要基于单次结果奖励进行策略优化与评估,导致初始生成质量与反馈驱动的修正能力混杂,丢失中间轮次的细粒度执行信号,且无法有效评估策略是否真正获得自修复能力。本文提出测试感知策略精炼(TaPR)框架,在统一多轮交互协议下,将执行反馈转化为每轮的测试通过率密集奖励。在LiveCodeBench的219个代码生成问题上,TaPR将整体三轮成功率(Pass@3)提升2.44个百分点。在预定义的7B/8B高容量子集上,准确率从30.25%提升至33.56%(+3.31个百分点),配对试验中42次提升、13次下降。在相同条件下的Qwen3-8B消融实验中,密集奖励在前10步均提供非零反馈,且在测试预算内达到比仅基于结果的GRPO更高的硬子集峰值性能,尽管后者在第300步时接近整体Pass@3水平。核心贡献在于奖励分解框架与轮次感知评估协议,实现首次生成质量与多轮修复能力的解耦。
原文摘要 · Abstract (English)
Multi-turn code agents rely on execution feedback to repair incorrect programs, yet standard reinforcement learning paradigms optimize and evaluate policy performance primarily using single-shot outcome rewards. This misalignment conflates initial code generation with feedback-driven refinement, discards granular execution signals across intermediate turns, and fails to evaluate whether the policy actually acquires self-repair capabilities. We propose Test-aware Policy Refinement (TaPR), a framework that transforms execution feedback into a dense per-turn test-pass-ratio reward under a consistent multi-turn interaction protocol. Across six models on 219 code-generation problems from LiveCodeBench, TaPR improves the pooled three-turn success rate (Pass@3) by 2.44 percentage points. In the predefined 7B/8B high-headroom slice, pooled accuracy increases from 30.25% to 33.56% (+3.31 pp), with 42 improvements and 13 regressions in paired trials. On a matched Qwen3-8B ablation, the dense reward supplies nonzero feedback in all of the first ten steps and reaches a higher Hard-subset peak than outcome-only GRPO within the tested budget, although GRPO nearly matches pooled Pass@3 by step 300. Our primary contribution is a reward-decomposition framework and a turn-aware evaluation protocol that decouple first-shot generation quality from multi-turn repair competence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。