用符号执行优化代码生成模型训练数据，提升代码质量判断能力。
Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLMs
- 用符号执行构建更全面的代码评估数据集，改进奖励模型训练。
- 奖励模型在代码质量评估上显著优于基线CodeRL，提升明显。
- 适合关注代码生成与强化学习结合的研究者与开发者。
代码生成大语言模型已成为现代软件开发的关键工具，可提升生产力并加速开发进程。本文研究通过强化学习和直接偏好优化对代码生成大模型进行微调，以进一步提升性能。为此，我们利用符号执行技术增强奖励模型的训练数据，确保数据更具全面性和客观性。通过符号执行构建的自定义数据集能更好捕捉代码评估中的细微差别。基于该数据集微调的奖励模型，在评估生成代码质量方面显著优于基线模型CodeRL。借助奖励模型反馈训练的代码生成模型，其表现与CodeRL基准相当。
原文摘要 · Abstract (English)
Code-generating Large Language Models (LLMs) have become essential tools in modern software development, enhancing productivity and accelerating development. This paper aims to investigate the fine-tuning of code-generating LLMs using Reinforcement Learning and Direct Preference Optimization, further improving their performance. To achieve this, we enhance the training data for the reward model with the help of symbolic execution techniques, ensuring more comprehensive and objective data. With symbolic execution, we create a custom dataset that better captures the nuances in code evaluation. Our reward models, fine-tuned on this dataset, demonstrate significant improvements over the baseline, CodeRL, in estimating the quality of generated code. Our code-generating LLMs, trained with the help of reward model feedback, achieve similar results compared to the CodeRL benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。