arXiv:2512.21919cs.CL2025-12被引 18

无需执行代码的奖励模型,让编程代理在测试和强化学习中表现更优。

SWE-RM: Execution-free Feedback For Software Engineering Agents

  • 用混合专家架构构建无需运行代码的奖励模型,提升反馈精度。
  • 在SWE-Bench Verified上使Qwen3-Coder模型准确率提升至62.0%~74.6%。
  • 适合追求高鲁棒性的开源编程代理研发者使用。

基于执行的反馈(如单元测试)广泛用于通过测试时扩展(TTS)和强化学习(RL)开发编码代理。该范式依赖可扩展且可靠的单元测试用例提供精确反馈,但反馈常稀疏,难以区分成功或失败轨迹。相比之下,来自奖励模型的无执行反馈无需依赖单元测试,可提供更细粒度信号。尽管潜力巨大,真实软件工程(SWE)代理的无执行反馈仍研究不足。我们发现,两个在TTS上表现相近的验证器在RL中却差异显著。直观上,TTS反映模型选择最优轨迹的能力,但该能力未必能泛化至RL。为此,我们识别出对RL训练至关重要的两个因素:分类准确性和校准性。通过系统控制实验,我们探究了训练数据规模、策略混合及数据源构成等因素的影响。基于此,提出SWE-RM,一个采用混合专家架构的奖励模型,总参数量30B,推理时激活3B。SWE-RM显著提升代理在TTS与RL上的性能,例如将Qwen3-Coder-Flash在SWE-Bench Verified上的准确率从51.6%提升至62.0%,将Qwen3-Coder-Max从67.0%提升至74.6%,达到开源模型新纪录。

原文摘要 · Abstract (English)

Execution-based feedback like unit testing is widely used in the development of coding agents through test-time scaling (TTS) and reinforcement learning (RL). This paradigm requires scalable and reliable collection of unit test cases to provide accurate feedback, and the resulting feedback is often sparse and cannot effectively distinguish between trajectories that are both successful or both unsuccessful. In contrast, execution-free feedback from reward models can provide more fine-grained signals without depending on unit test cases. Despite this potential, execution-free feedback for realistic software engineering (SWE) agents remains underexplored. Aiming to develop versatile reward models that are effective across TTS and RL, however, we observe that two verifiers with nearly identical TTS performance can nevertheless yield very different results in RL. Intuitively, TTS primarily reflects the model's ability to select the best trajectory, but this ability does not necessarily generalize to RL. To address this limitation, we identify two additional aspects that are crucial for RL training: classification accuracy and calibration. We then conduct comprehensive controlled experiments to investigate how to train a robust reward model that performs well across these metrics. In particular, we analyze the impact of various factors such as training data scale, policy mixtures, and data source composition. Guided by these investigations, we introduce SWE-RM, an accurate and robust reward model adopting a mixture-of-experts architecture with 30B total parameters and 3B activated during inference. SWE-RM substantially improves SWE agents on both TTS and RL performance. For example, it increases the accuracy of Qwen3-Coder-Flash from 51.6% to 62.0%, and Qwen3-Coder-Max from 67.0% to 74.6% on SWE-Bench Verified using TTS, achieving new state-of-the-art performance among open-source models.

编程代理奖励模型强化学习代码生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。