实证发现代码审查机器人评价难，人工标签受工作流影响大。
Understanding the Limits of Automated Evaluation for Code Review Bots in Practice
- 用大模型自动评机器人评论，对比工程师标注结果
- 不同模型和评分方式下，准确率仅44%~62%
- 适合关注工业场景评测局限性的研发团队
自动化代码审查（ACR）机器人在工业软件开发中日益普及，用于辅助开发者评审代码提交。随着应用扩展，如何可靠、规模化评估机器人生成评论的价值成为关键挑战。实践中，常依赖开发者行为与标注，但这些信号受上下文和组织因素影响，难以作为客观基准。本文基于Beko公司2,604条机器人评论数据，分析了基于大模型的自动化评估方法在工业场景中的可行性与局限性。采用G-Eval与LLM-as-a-Judge两种策略，分别以二分类（fixed/wontFix）和0-4级李克特量表进行评估，与工程师标注对比。结果显示，不同模型（Gemini-2.5-pro、GPT-4.1-mini、GPT-5.2）在两类评估方式下的一致性仅介于0.44至0.62之间，且表现随模型与设计方式变化明显。研究揭示，开发者对评论的处理行为不仅反映质量，还受工作优先级、流程压力等复杂因素影响，难以通过静态数据完全捕捉。访谈一位软件工程总监进一步确认，标签行为受组织约束显著，强化了将此类信号视为客观标准的困难。
原文摘要 · Abstract (English)
Automated code review (ACR) bots are increasingly used in industrial software development to assist developers during pull request (PR) review. As adoption grows, a key challenge is how to evaluate the usefulness of bot-generated comments reliably and at scale. In practice, such evaluation often relies on developer actions and annotations that are shaped by contextual and organizational factors, complicating their use as objective ground truth. We examine the feasibility and limitations of automating the evaluation of LLM-powered ACR bots in an industrial setting. We analyze an industrial dataset from Beko comprising 2,604 bot-generated PR comments, each labeled by software engineers as fixed/wontFix. Two automated evaluation approaches, G-Eval and an LLM-as-a-Judge pipeline, are applied using both binary decisions and a 0-4 Likert-scale formulation, enabling a controlled comparison against developer-provided labels. Across Gemini-2.5-pro, GPT-4.1-mini, and GPT-5.2, both evaluation strategies achieve only moderate alignment with human labels. Agreement ratios range from approximately 0.44 to 0.62, with noticeable variation across models and between binary and Likert-scale formulations, indicating sensitivity to both model choice and evaluation design. Our findings highlight practical limitations in fully automating the evaluation of ACR bot comments in industrial contexts. Developer actions such as resolving or ignoring comments reflect not only comment quality, but also contextual constraints, prioritization decisions, and workflow dynamics that are difficult to capture through static artifacts. Insights from a follow-up interview with a software engineering director further corroborate that developer labeling behavior is strongly influenced by workflow pressures and organizational constraints, reinforcing the challenges of treating such signals as objective ground truth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。