用多个AI评委协作打分，让自动评估更接近人工判断。
An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks
- 设计五种不同评分策略的AI评委，动态组合成团队打分。
- 在代码生成等任务中，与人工评价相关性提升29.6%至140.8%。
- 适合需要高效准确评估生成代码的开发者和研究者使用。
大型语言模型（LLMs）在生成代码片段、补丁和注释等软件工程任务中日益普及，但准确评估其生成结果的正确性仍是重大挑战。人工评价虽准确但耗时且难以扩展；现有自动评估指标虽可扩展性强，却常无法真实反映生成内容的正确性。本文提出SE-Jury，首个专为软件工程任务设计的LLM集成评审评估指标。SE-Jury定义五种独立评审策略，每种由一个独立判官实现，并通过动态团队选择机制，选出最适配的判官组合进行集成评分。我们在涵盖代码生成、程序修复和代码摘要三类主流任务的多个基准上评估了SE-Jury。结果表明，该方法在各项任务中均显著提升与人工评价的相关性，提升幅度达29.6%至140.8%。在代码生成与程序修复任务中，其与人工标注者的一致性接近人工标注者间的互评一致性。这些发现证明SE-Jury在软件工程任务中具备高可扩展性与可靠性，是人工评价的有力替代方案。
原文摘要 · Abstract (English)
Large Language Models (LLMs) and other automated techniques have been increasingly used to support software developers by generating software artifacts such as code snippets, patches, and comments. However, accurately assessing the correctness of these generated artifacts remains a significant challenge. On one hand, human evaluation provides high accuracy but is labor-intensive and lacks scalability. On the other hand, many automatic evaluation metrics are scalable and require minimal human effort, but they often fail to accurately reflect the actual correctness of generated software artifacts. In this paper, we present SE-Jury, the first evaluation metric for LLM-as-Ensemble-Judge specifically designed to accurately assess the correctness of generated software artifacts. SE-Jury first defines five distinct evaluation strategies, each implemented by an independent judge. A dynamic team selection mechanism then identifies the most appropriate subset of judges as a team to produce a final correctness score through ensembling. We evaluate SE-Jury across a diverse set of software engineering (SE) benchmarks that span three popular SE tasks: code generation, automated program repair, and code summarization. Results demonstrate that SE-Jury consistently achieves a higher correlation with human judgments, with improvements ranging from 29.6% to 140.8% over existing automatic metrics. SE-Jury reaches agreement levels with human annotators that are close to inter-annotator agreement in code generation and program repair. These findings underscore SE-Jury's potential as a scalable and reliable alternative to human evaluation in these SE tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。