让大模型从错误步骤中学习，提升代码生成成功率
Step Rejection Fine-Tuning: A Practical Distillation Recipe
- 用批判模型评估每步代码正确性，保留错误但屏蔽损失
- 在SWE-bench上将修复率提升至32.2%，比传统方法高1.3个百分点
- 适合训练需要容错与纠错能力的代码生成模型
拒绝微调（RFT）是训练大语言模型代理的标准方法，即丢弃失败的执行轨迹。在SWE-bench任务中，这意味着过滤掉提交补丁未通过测试的运行。然而，该方法会丢弃未解决的轨迹，而这些轨迹在困难任务中占很大比例，且可能部分正确。本文提出步骤拒绝微调（SRFT），一种利用这些未解决轨迹的实用方法。我们使用一个批判性LLM评估轨迹中每一步的正确性，训练时对错误步骤屏蔽损失，但保留在上下文窗口中。这样模型能学习从错误中恢复而不重复错误。在SWE-bench Verified上的评估显示，虽然RFT通过排除未解决轨迹将解决率提升了2.4%，但SRFT通过筛选而非完全丢弃轨迹，将解决率提升至3.7%，达到总解决率32.2%。
原文摘要 · Abstract (English)
Rejection Fine-Tuning (RFT) is a standard method for training LLM agents, where unsuccessful trajectories are discarded from the training set. In the context of SWE-bench tasks, this corresponds to filtering out runs where the submitted patch does not pass the tests. However, this approach discards unresolved trajectories, even though they form a large portion of all trajectories for hard tasks and even then may be partially correct. In this work, we propose Step Rejection Fine-Tuning (SRFT) - a practical way to leverage these unresolved trajectories. For this, we employ a critic LLM to assess the correctness of each step in a trajectory. Consequently, during training, we mask the loss for erroneous steps while retaining them in the context window. This way we ensure the model learns to recover from errors without reproducing them. Evaluation on SWE-bench Verified shows that while RFT improves the resolution rate by 2.4% by excluding unresolved trajectories, SRFT improves it by 3.7% by filtering them instead of discarding completely, reaching the total resolution rate of 32.2%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。