用少量正确代码示例提升大模型生成代码的正确性评估效果
In-Context Learning as an Effective Estimator of Functional Correctness of LLM-Generated Code
- 通过上下文学习，用少量正确代码样例增强代码质量评估
- 在无测试用例时，显著提升代码正确性预测准确率
- 适合快速开发中需快速验证生成代码质量的开发者
在采用特征驱动或快速应用开发模式的软件项目中，缺乏测试用例时难以评估大模型生成代码的功能正确性。如同用户从检索结果列表中选择相关文档，代码生成流程也需要开发者从按后验概率排序的多个候选方案中选出最优解。这要求评估排序列表的整体质量——类似于信息检索中的查询性能预测（QPP）。本文提出一种基于上下文学习（ICL）的代码质量评估方法，实验表明：在训练集中提供少量功能正确的代码示例，可有效提升现有QPP方法及零样本代码质量评估的效果。
原文摘要 · Abstract (English)
When applying LLM-based code generation to software development projects that follow a feature-driven or rapid application development approach, it becomes necessary to estimate the functional correctness of the generated code in the absence of test cases. Just as a user selects a relevant document from a ranked list of retrieved ones, a software generation workflow requires a developer to choose (and potentially refine) a generated solution from a ranked list of alternative solutions, ordered by their posterior likelihoods. This implies that estimating the quality of a ranked list -- akin to estimating "relevance" for query performance prediction (QPP) in IR -- is also crucial for generative software development, where quality is defined in terms of "functional correctness". In this paper, we propose an in-context learning (ICL) based approach for code quality estimation. Our findings demonstrate that providing few-shot examples of functionally correct code from a training set enhances the performance of existing QPP approaches as well as a zero-shot-based approach for code quality estimation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。