无需参考文本，精准评估代码摘要的事实一致性。
ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization
- 基于代码段落级事实不一致标准，结合依赖关系分析。
- 在真实代码摘要上与人工判断相关性最高，优于之前最优方法15%-18%。
- 适合需要高精度评估代码生成质量的研究者和开发者。
随着大语言模型（LLMs）生成长篇描述性代码摘要能力的提升，准确可靠的真相一致性评估成为关键挑战。然而，以往方法主要针对孤立代码片段的简短摘要设计，难以对多句功能进行细粒度评估，且无法有效衡量真实代码摘要中常见的依赖上下文。为此，我们提出 ReFEree，一种无需参考文本、可细粒度评估真实世界代码摘要事实一致性的方法。我们定义了适用于代码摘要的事实不一致标准，并利用这些标准与依赖信息在段落级别进行评估，再将结果聚合为细粒度得分。我们构建了一个带有人工标注事实一致性标签的代码摘要基准数据集。评估结果显示，ReFEree 在13个基线中与人工判断的相关性最高，相比先前最先进方法提升15-18%。代码与数据已开源：https://github.com/bsy99615/ReFEree.git。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) have become capable of generating long and descriptive code summaries, accurate and reliable evaluation of factual consistency has become a critical challenge. However, previous evaluation methods are primarily designed for short summaries of isolated code snippets. Consequently, they struggle to provide fine-grained evaluation of multi-sentence functionalities and fail to accurately assess dependency context commonly found in real-world code summaries. To address this, we propose ReFEree, a reference-free and fine-grained method for evaluating factual consistency in real-world code summaries. We define factual inconsistency criteria specific to code summaries and evaluate them at the segment level using these criteria along with dependency information. These segment-level results are then aggregated into a fine-grained score. We construct a code summarization benchmark with human-annotated factual consistency labels. The evaluation results demonstrate that ReFEree achieves the highest correlation with human judgment among 13 baselines, improving 15-18% over the previous state-of-the-art. Our code and data are available at https://github.com/bsy99615/ReFEree.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。