用偏好优化提升代码大模型的对齐效果，效果优于传统方法。
Aligning CodeLLMs with Direct Preference Optimization
- 采用DPO算法，仅需偏好数据对即可实现细粒度奖励
- 在MBPP和HumanEval上显著提升现有代码模型表现
- 适合关注代码生成质量与对齐优化的研究者
过去一年，大语言模型在多个领域快速进展。其中代码大模型（CodeLLMs）因能辅助编程任务并体现逻辑推理能力而备受关注。然而，当前多数工作聚焦预训练与监督微调，忽视了对齐阶段这一后训练关键环节。本文首次指出，常用强化学习算法PPO在代码模型对齐中表现不佳，因其奖励规则粗粒度且可能有误。为此提出采用直接偏好优化（DPO）算法，仅依赖偏好数据对即可自动排序，形成更鲁棒的细粒度奖励机制。同时构建了用于代码大模型的DPO偏好数据收集流程。实验表明，该方法显著提升现有CodeLLMs在MBPP和HumanEval基准上的性能。
原文摘要 · Abstract (English)
The last year has witnessed the rapid progress of large language models (LLMs) across diverse domains. Among them, CodeLLMs have garnered particular attention because they can not only assist in completing various programming tasks but also represent the decision-making and logical reasoning capabilities of LLMs. However, current CodeLLMs mainly focus on pre-training and supervised fine-tuning scenarios, leaving the alignment stage, which is important for post-training LLMs, under-explored. This work first identifies that the commonly used PPO algorithm may be suboptimal for the alignment of CodeLLM because the involved reward rules are routinely coarse-grained and potentially flawed. We then advocate addressing this using the DPO algorithm. Based on only preference data pairs, DPO can render the model rank data automatically, giving rise to a fine-grained rewarding pattern more robust than human intervention. We also contribute a pipeline for collecting preference pairs for DPO on CodeLLMs. Studies show that our method significantly improves the performance of existing CodeLLMs on benchmarks such as MBPP and HumanEval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。