用多人主观评分法评估大模型代码审查质量，发现其表现优于人类。
On Assessing the Relevance of Code Reviews Authored by Generative Models
- 设计多主观排序评估法，对比多个真人评审对AI生成评论的打分。
- 在280个请求上测试，ChatGPT评论得分显著高于平台优质人工回复。
- 适合关注AI辅助开发、代码质量保障的研究者与工程师参考。
大型语言模型如ChatGPT在代码审查中虽能提升效率，但其正确性与安全性仍存疑。现有评估方法或依赖单一基准进行自动比对，无法体现人类观点的多样性；或采用模糊的“有用性”主观判断。本文提出基于多主观排序的新评估方法：利用包含280个独立代码审查请求及来自CodeReview StackExchange的对应评论的数据集，多位人类评审者对ChatGPT生成的评论与平台优质人工回复进行质量排序。结果表明，ChatGPT生成的评论在多数情况下得分更高，甚至超越被采纳的人工答案。该方法不仅推动了生成式AI在代码审查中更合理的评估，也警示了未经审慎验证就直接集成的风险。
原文摘要 · Abstract (English)
The use of large language models like ChatGPT in code review offers promising efficiency gains but also raises concerns about correctness and safety. Existing evaluation methods for code review generation either rely on automatic comparisons to a single ground truth, which fails to capture the variability of human perspectives, or on subjective assessments of "usefulness", a highly ambiguous concept. We propose a novel evaluation approach based on what we call multi-subjective ranking. Using a dataset of 280 self-contained code review requests and corresponding comments from CodeReview StackExchange, multiple human judges ranked the quality of ChatGPT-generated comments alongside the top human responses from the platform. Results show that ChatGPT's comments were ranked significantly better than human ones, even surpassing StackExchange's accepted answers. Going further, our proposed method motivates and enables more meaningful assessments of generative AI's performance in code review, while also raising awareness of potential risks of unchecked integration into review processes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。