用函数级执行反馈优化代码生成，提升模型对正确代码的偏好。
Function-Level Execution Feedback for Code Preference Optimization
- 将代码步骤定义为模块级函数，通过单元测试自动标注正确性。
- 在HumanEval(+)等4个数据集上优于仅基于结果的优化方法。
- 执行反馈比人工判断更准确，避免错误标签污染训练过程。
过程监督在数学推理中已显著提升性能，但代码生成中仍缺乏标准的‘步骤’概念。现有方法针对行、推理轨迹或程序状态进行监督，难以确定标注对象。本文提出STEP-KTODER框架，将多函数程序中的模块级函数视为步骤，并通过自动生成的单元测试赋予二元正确性标签。该方法实现了面向代码的分步KTO，结合函数级过程监督与全程序结果反馈。在HumanEval(+), MBPP(+), BigCodeBench, LiveCodeBench上评估显示，其性能优于仅依赖结果的KTO和DPO。进一步分析表明，基于执行的标签至关重要：大模型作为评判者会系统性高估函数失败率，导致正向步骤标签被污染，进而损害下游偏好优化效果。代码开源于https://github.com/inechnech/STEP-KTODER。
原文摘要 · Abstract (English)
Process supervision has improved mathematical reasoning, where intermediate steps are naturally expressed as chains of thought. In code generation, however, process supervision remains underexplored because there is no standard notion of a step. Supervision can target lines, reasoning traces, or program states, making it unclear what to label and optimize. We propose STEP-KTODER, a framework for code preference optimization that defines steps as module-level functions in decomposed multi-function programs and assigns binary correctness labels via automatically generated unit tests. Our method provides a code-specific instantiation of stepwise KTO, combining function-level process supervision with outcome-level feedback on the full program. We evaluate on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench, showing that STEP-KTODER improves over outcome-only KTO and DPO. Further analysis shows that execution-based labels are essential: LLM-as-a-judge annotations systematically over-predict function failures, corrupt positive step labels, and degrade downstream preference optimization. Code is available at: https://github.com/inechnech/STEP-KTODER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。