用大模型评分代码摘要，比传统方法更准且能结合代码上下文。
Simple and Effective Baselines for Code Summarisation Evaluation
- 让大模型综合代码内容给出摘要评分，不依赖参考摘要。
- 在多个数据集上表现优于或媲美现有自动评估指标。
- 适合评估代码库文档质量，也可用于其他生成任务评估。
代码注释很有用，但编写耗时。尽管出现了多种生成代码摘要的技术，但对比它们仍困难，因为人工评估成本高，而自动指标不可靠。本文提出一种简单的新基线：让大语言模型（LLM）对摘要整体打分。与基于n-gram和嵌入的方法不同，该方法能结合代码内容进行评分。我们还设计了不依赖参考摘要的变体，可用于评估代码库中的文档质量等其他任务。实验表明，该方法效果与或优于现有指标，但建议与嵌入式方法结合使用，以降低大模型固有偏见风险。
原文摘要 · Abstract (English)
Code documentation is useful, but writing it is time-consuming. Different techniques for generating code summaries have emerged, but comparing them is difficult because human evaluation is expensive and automatic metrics are unreliable. In this paper, we introduce a simple new baseline in which we ask an LLM to give an overall score to a summary. Unlike n-gram and embedding-based baselines, our approach is able to consider the code when giving a score. This allows us to also make a variant that does not consider the reference summary at all, which could be used for other tasks, e.g., to evaluate the quality of documentation in code bases. We find that our method is as good or better than prior metrics, though we recommend using it in conjunction with embedding-based methods to avoid the risk of LLM-specific bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。