用大模型自动分类代码评审意见，效果优于传统方法。
Exploring the Potential of Large Language Models in Fine-Grained Review Comment Classification
- 直接用大模型分类17类代码评审意见，无需大量人工标注。
- 在高频和低频类别上表现均衡，尤其在五类关键意见上准确率更高。
- 适合需要快速部署、数据少的代码审查分析场景。
代码评审是软件开发中的关键环节，但如今评审内容繁杂，常包含琐碎问题。已有研究尝试用自动化方法对评审意见进行分类，以评估评审效果。然而，以往工作多依赖需大量人工标注的监督学习模型。为解决这一局限，我们探索大语言模型（LLMs）在细粒度代码评审意见分类中的潜力。我们评估了LLMs对17类评审意见的分类性能。结果表明，LLMs能有效分类评审意见，其整体表现优于使用深度学习模型的最先进方法。尤其在五个最具价值的类别上，LLMs表现更优，而现有方法因训练样本不足难以应对。与依赖特定小规模训练数据分布的方法不同，我们的结果表明LLMs在高频率和低频率类别间表现出更均衡的性能。这表明，大模型可为代码审查分析提供可扩展的解决方案，有助于提升代码评审效率。
原文摘要 · Abstract (English)
Code review is a crucial practice in software development. As code review nowadays is lightweight, various issues can be identified, and sometimes, they can be trivial. Research has investigated automated approaches to classify review comments to gauge the effectiveness of code reviews. However, previous studies have primarily relied on supervised machine learning, which requires extensive manual annotation to train the models effectively. To address this limitation, we explore the potential of using Large Language Models (LLMs) to classify code review comments. We assess the performance of LLMs to classify 17 categories of code review comments. Our results show that LLMs can classify code review comments, outperforming the state-of-the-art approach using a trained deep learning model. In particular, LLMs achieve better accuracy in classifying the five most useful categories, which the state-of-the-art approach struggles with due to low training examples. Rather than relying solely on a specific small training data distribution, our results show that LLMs provide balanced performance across high- and low-frequency categories. These results suggest that the LLMs could offer a scalable solution for code review analytics to improve the effectiveness of the code review process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。