arXiv:2508.03501cs.LGcs.CL2025-08被引 28

用强化学习训练能持续交互的编程智能体,性能提升超3倍。

Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning

  • 先用执行反馈微调,再用同步强化学习迭代优化。
  • 在SWE-bench上通过率从11%提至39%,超过20%基线。
  • 适合想用开源模型做复杂编程任务的研究者和开发者。

强化学习在大语言模型中的应用多集中于单轮问题,如数学推理或一次性代码生成。然而,真实软件工程场景需要与状态化环境进行丰富多轮交互,每一步动作都会带来非平凡的反馈。为弥合这一差距,本文展示了强化学习在该通用范式中的成功应用。方法包括:首先使用执行反馈进行拒绝微调(RFT),使模型有效遵循指令并正确格式化;随后采用同步强化学习管道,结合DAPO进行迭代优化。将该流程应用于Qwen2.5-72B-Instruct,在SWE-bench Verified基准上其Pass@1从11%提升至39%,显著优于20%的RFT基线。在SWE-rebench的5月和6月数据集上,通过率分别达到35%和31%,已具备与更大型模型(如DeepSeek-V3-0324、Qwen3-235B-A22B)相当的竞争力,证明了该方法在使用开源权重模型训练多轮交互智能体方面的实用性。

原文摘要 · Abstract (English)

Research on applications of reinforcement learning (RL) to large language models has mostly been focused on single-turn problems, such as mathematical reasoning or single-shot code generation. While these problems can be viewed as token-level multi-turn Markov decision processes (MDPs), this view corresponds to a degenerate case of multi-turn interaction where the environment provides no feedback. This contrasts with many real-world domains, such as software engineering (SWE), which require rich multi-turn interactions with a stateful environment that responds to each action with a non-trivial observation. To bridge this gap, we demonstrate the successful application of RL to this general regime. Our methodology begins with rejection fine-tuning (RFT) using execution feedback to train a policy to follow instructions and formatting effectively, followed by a synchronous RL pipeline using DAPO for iterative improvement. Applying this pipeline to Qwen2.5-72B-Instruct, we increase its Pass@1 on the SWE-bench Verified benchmark from 11% to 39%, substantially improving upon the 20% RFT baseline. On the May and June splits of SWE-rebench, the resulting agent achieves Pass@1 of 35% and 31% respectively, competitive with even larger models such as DeepSeek-V3-0324 or Qwen3-235B-A22B, demonstrating that our methodology offers a practical approach for training capable agents for multi-turn interactive tasks using open-weight models.

强化学习编程智能体多轮交互开源模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。