用自动提炼法构建代码审查评论数据集,提升大模型精准发现代码问题的能力。
Distilling Desired Comments for Enhanced Code Review with Large Language Models
- 从现有评审数据中自动识别出目标评论,生成高质量训练数据
- 在15万+条数据上达到88.93%准确率,显著优于现有方法
- 适用于想改进代码审查能力的开发者或研究者
近年来,大型语言模型(LLMs)在代码理解方面表现优异,被广泛用于代码审查。然而,现有基于LLM的方案在生成目标审查评论(DRCs)时效果不佳,主要因幻觉等问题。为提升其能力,需使用包含大量真实DRCs的定制数据集进行微调,但此类数据集尚未存在,且人工标注成本过高。本文提出一种数据集提炼方法Desiview,可自动从代码审查数据集中提取出有效的DRCs,构建精炼数据集。在包含超过15万条评审记录的CodeReviewer数据集上,Desiview取得88.93%精度、80.37%召回率、86.67%准确率和84.44%F1值,优于现有最优方法。我们进一步以最新版LLaMA系列(LLaMA 3与LLaMA 3.1)为基础,通过该数据集微调得到Desiview4FT,并引入KTO对齐机制,将非DRC评论也纳入训练,形成Desiview4FA。验证结果表明,Desiview4FA略优于Desiview4FT,二者均显著优于基础模型。人工评估确认,新模型更准确地识别问题,生成的评论也更清晰地描述代码缺陷。
原文摘要 · Abstract (English)
There has been a growing interest in using Large Language Models (LLMs) for code review thanks to their proven proficiency in code comprehension. The primary objective of most review scenarios is to generate desired review comments (DRCs) that explicitly identify issues to trigger code fixes. However, existing LLM-based solutions are not so effective in generating DRCs for various reasons such as hallucination. To enhance their code review ability, they need to be fine-tuned with a customized dataset that is ideally full of DRCs. Nevertheless, such a dataset is not yet available, while manual annotation of DRCs is too laborious to be practical. In this paper, we propose a dataset distillation method, Desiview, which can automatically construct a distilled dataset by identifying DRCs from a code review dataset. Experiments on the CodeReviewer dataset comprising more than 150K review entries show that Desiview achieves an impressive performance of 88.93%, 80.37%, 86.67%, and 84.44% in terms of Precision, Recall, Accuracy, and F1, respectively, surpassing state-of-the-art methods. To validate the effect of such a distilled dataset on enhancing LLMs' code review ability, we first fine-tune the latest LLaMA series (i.e., LLaMA 3 and LLaMA 3.1) to build model Desiview4FT. We then enhance the model training effect through KTO alignment by feeding those review comments identified as non-DRCs to the LLMs, resulting in model Desiview4FA. Verification results indicate that Desiview4FA slightly outperforms Desiview4FT, while both models have significantly improved against the base models in terms of generating DRCs. Human evaluation confirms that both models identify issues more accurately and tend to generate review comments that better describe the issues contained in the code than the base LLMs do.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。