用对比学习评估代码与任务意图匹配度，无需参考代码。
MATCH: Task-Driven Code Evaluation through Contrastive Learning
- 通过对比学习生成代码与任务描述的语义嵌入
- 在多语言中与功能正确性和人工偏好相关性更强
- 适合无参考代码时的自动化代码质量评估
基于AI的代码生成日益普遍，据估计GitHub Copilot已生成GitHub上46%的代码。准确评估生成代码与开发者意图的一致性仍是关键挑战。传统方法如单元测试难以扩展且成本高；语法相似性指标（如BLEU、ROUGE）无法捕捉代码功能，CodeBERTScore等需参考代码，而参考代码并不总可用。为填补无参考评估的空白，现有替代方案如ICE-Score有限，本文提出MATCH，一种新型无参考评估指标。MATCH利用对比学习生成代码与自然语言任务描述的有意义嵌入，实现反映代码是否正确实现任务的相似性评分。实验表明，MATCH在多种编程语言中与功能正确性和人工偏好相关性优于现有指标。
原文摘要 · Abstract (English)
AI-based code generation is increasingly prevalent, with GitHub Copilot estimated to generate 46% of the code on GitHub. Accurately evaluating how well generated code aligns with developer intent remains a critical challenge. Traditional evaluation methods, such as unit tests, are often unscalable and costly. Syntactic similarity metrics (e.g., BLEU, ROUGE) fail to capture code functionality, and metrics like CodeBERTScore require reference code, which is not always available. To address the gap in reference-free evaluation, with few alternatives such as ICE-Score, this paper introduces MATCH, a novel reference-free metric. MATCH uses Contrastive Learning to generate meaningful embeddings for code and natural language task descriptions, enabling similarity scoring that reflects how well generated code implements the task. We show that MATCH achieves stronger correlations with functional correctness and human preference than existing metrics across multiple programming languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。