用评分标准提升代码生成智能体的中间行为质量
Beyond Verifiable Rewards: Rubric-Based GRM for Reinforced Fine-Tuning SWE Agents
- 设计基于人工评分标准的生成式奖励模型，指导中间步骤优化
- 在多步任务中显著降低错误行为比例，最终测试通过率提升
- 适合需要高质量推理过程的代码生成与自动化调试场景
尽管大语言模型在软件工程任务中取得进展，但端到端微调仍依赖可验证的终态奖励（如单元测试是否通过）。这类二元信号虽能判断结果正确性，却难以指导多步交互中的中间行为，限制了整体求解质量。为此，我们提出基于评分标准的生成式奖励模型（GRM），引入人工设计的评判准则，以鼓励或抑制特定行为模式，并通过轨迹筛选构建高质量训练数据。在软件工程任务的强化微调中，该方法优于仅依赖终态分数的拒绝采样：能更有效抑制不良行为、促进有益行为，经案例分析验证，并显著提升最终测试准确率。
原文摘要 · Abstract (English)
Despite recent progress in Large Language Model (LLM) Agents for Software Engineering (SWE) tasks, end-to-end fine-tuning typically relies on verifiable terminal rewards such as whether all unit tests pass. While these binary signals reflect whether the final solution is correct, they provide little guidance for shaping intermediate behaviors during multi-step interactions, thereby limiting improvements in the overall quality of the resolution process. To address this, we introduce a rubric-based Generative Reward Model (GRM) that provides richer learning signals. The GRM is equipped with human-designed rubrics that indicate criteria for encouraging or discouraging specific behavioral patterns, and we leverage this feedback for high-quality training data collection via trajectory filtration. When used for Reinforced Fine-Tuning (RFT) on SWE Tasks, our approach outperforms terminal-score-only rejection sampling: it more effectively suppresses undesirable patterns while promoting beneficial ones, as confirmed by case analyses, and it ultimately improves final test accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。