用分步强化微调提升开源大模型的代码问题修复能力
SoRFT: Issue Resolving with Subtask-oriented Reinforced Fine-Tuning
- 将问题修复拆解为定位文件、函数、行和生成代码四步
- 在SWE-Bench Verified上达到21.4%的修复率，领先开源模型
- 无需依赖商业模型，适合想自建代码修复系统的团队
主流问题修复框架多依赖商业模型，导致成本高且存在隐私风险。现有训练方法泛化能力差，未能充分利用开源开发资源。我们提出面向子任务的强化微调（SoRFT），通过结构化拆解问题修复流程：文件定位、函数定位、行定位和代码修改生成。SoRFT包含两个阶段：(1) 基于拒绝采样的监督微调，利用真实答案过滤思维链数据后再微调大模型；(2) 基于规则的强化学习，采用基于真实答案奖励的PPO算法。在SWE-Bench Verified和SWE-Bench Lite上评估表明，训练后的模型在开源模型中达到最先进水平（如SoRFT-Qwen-7B在SWE-Bench Verified上解决21.4%的问题）。实验结果证明，SoRFT显著提升修复性能与泛化能力，提供低成本替代商业模型的方案。
原文摘要 · Abstract (English)
Mainstream issue-resolving frameworks predominantly rely on commercial models, leading to high costs and privacy concerns. Existing training approaches for issue resolving struggle with poor generalization and fail to fully leverage open-source development resources. We propose Subtask-oriented Reinforced Fine-Tuning (SoRFT), a novel training approach to enhance the issue resolving capability of LLMs. We decomposes issue resolving into structured subtasks: file localization, function localization, line localization, and code edit generation. SoRFT consists of two training stages: (1) rejection-sampled supervised fine-tuning, Chain of Thought (CoT) data is filtered using ground-truth before fine-tuning the LLM, and (2) rule-based reinforcement learning, which leverages PPO with ground-truth based rewards. We evaluate the SoRFT-trained model on SWE-Bench Verified and SWE-Bench Lite, achieving state-of-the-art (SOTA) performance among open-source models (e.g., resolve 21.4% issues on SWE-Bench Verified with SoRFT-Qwen-7B). The experimental results demonstrate that SoRFT significantly enhances issue-resolving performance, improves model generalization, and provides a cost-efficient alternative to commercial models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。