用大模型自评代码正确性，无需测试用例也能可靠判断。
CodeJudge: Evaluating Code Generation with Large Language Models
- 让大模型进行深度思考，提升代码语义评估准确性
- 在多个数据集和语言上表现优于现有方法
- 小模型也能超越大模型，适合资源有限的开发者
大语言模型在代码生成方面表现出色，但如何可靠评估其生成的代码仍是一个未解难题。本文提出CodeJudge，一种基于大模型的代码评估框架，可在无需测试用例的情况下评估生成代码的语义正确性。我们探索了多种引导大模型进行‘慢思考’的方法，以实现深入且可靠的评估。在四个代码生成数据集和五种编程语言上，使用四种大模型作为评估器进行了实验。结果表明，CodeJudge在大多数设置下显著优于现有方法。此外，与基于GPT-3.5的最先进评估方法相比，即使使用更小的Llama-3-8B-Instruct模型，CodeJudge也取得了更好效果。代码和数据集已开源于GitHub：https://github.com/VichyTong/CodeJudge。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown promising performance in code generation. However, how to reliably evaluate code generated by LLMs remains an unresolved problem. This paper presents CodeJudge, a code evaluation framework that leverages LLMs to evaluate the semantic correctness of generated code without the need for test cases. We investigate different ways to guide the LLM in performing "slow thinking" to arrive at an in-depth and reliable evaluation. We experimented with four LLMs as evaluators on four code generation datasets and five programming languages. The results show that CodeJudge significantly outperformed existing methods in most settings. Furthermore, compared with a SOTA GPT-3.5-based code evaluation method, CodeJudge achieved better results even when using a much smaller model, Llama-3-8B-Instruct. Our code and datasets are available on GitHub https://github.com/VichyTong/CodeJudge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。