提出无需参考文本的代码评审质量评估方法，更贴近人工判断。
CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells
- 基于大模型与静态分析器发现的代码问题和主张，构建评估框架。
- 在2.9千条评论上测试，与人工评分相关性达0.54（最高开源指标）。
- 适合研究自动化代码评审或开发评测工具的开发者使用。
自动化代码评审近年来受到机器学习领域广泛关注。然而，现有评论评估指标依赖于人类编写的参考评论（针对特定代码变更，即diff）。此外，代码评审属于一对多任务，同一代码变更存在多个有效评审意见。为此，我们提出CRScore——一种无需参考文本的评估指标，用于衡量评审质量的简洁性、全面性和相关性等维度。该方法将评估建立在大语言模型与静态分析器所识别出的代码主张与潜在问题基础上。实验表明，CRScore能生成有效且细粒度的评审质量评分，在开源指标中与人工判断的相关性最高（斯皮尔曼相关系数0.54），且比依赖参考的指标更敏感。我们还发布了包含2.9千条人工标注评分的数据集，涵盖机器生成与GitHub评论，以支持自动化评估方法的发展。
原文摘要 · Abstract (English)
The task of automated code review has recently gained a lot of attention from the machine learning community. However, current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). Furthermore, code review is a one-to-many problem, like generation and summarization, with many "valid reviews" for a diff. Thus, we develop CRScore - a reference-free metric to measure dimensions of review quality like conciseness, comprehensiveness, and relevance. We design CRScore to evaluate reviews in a way that is grounded in claims and potential issues detected in the code by LLMs and static analyzers. We demonstrate that CRScore can produce valid, fine-grained scores of review quality that have the greatest alignment with human judgment among open source metrics (0.54 Spearman correlation) and are more sensitive than reference-based metrics. We also release a corpus of 2.9k human-annotated review quality scores for machine-generated and GitHub review comments to support the development of automated metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。