构建代码纠错反馈评估环境，提升大模型代码修改能力。
Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code
- 用人类编辑轨迹和机器反馈构建数据集，支持强化学习训练。
- 新奖励函数使模型反馈效果优于GPT-4，代码通过率显著提升。
- 适合研究代码生成与智能编程助手的开发者使用。
本文提出Coffee-Gym，一个用于训练代码纠错反馈模型的强化学习环境。该环境包含两个核心组件：(1) Coffee数据集，收录了人类对编程问题的代码修改轨迹及针对错误代码的机器反馈；(2) CoffeeEval奖励函数，通过评估修正后代码在单元测试中的表现，真实反映反馈的有用性。Coffee-Gym解决了高质量训练数据稀缺的问题，其奖励机制优于当前最优模型（GPT-4）。基于该环境训练的反馈模型，在提升开源代码大模型的代码编辑能力方面表现更优，达到与闭源模型相当的水平。相关数据集与模型检查点已公开。
原文摘要 · Abstract (English)
This paper presents Coffee-Gym, a comprehensive RL environment for training models that provide feedback on code editing. Coffee-Gym includes two major components: (1) Coffee, a dataset containing humans' code edit traces for coding questions and machine-written feedback for editing erroneous code; (2) CoffeeEval, a reward function that faithfully reflects the helpfulness of feedback by assessing the performance of the revised code in unit tests. With them, Coffee-Gym addresses the unavailability of high-quality datasets for training feedback models with RL, and provides more accurate rewards than the SOTA reward model (i.e., GPT-4). By applying Coffee-Gym, we elicit feedback models that outperform baselines in enhancing open-source code LLMs' code editing, making them comparable with closed-source LLMs. We make the dataset and the model checkpoint publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。