测试五种提示技巧对代码生成正确率与质量的影响，发现组合使用未必更好。
The Impact of Prompt Programming on Function-Level Code Generation
- 构建7072条提示数据集，评估五类提示技巧对代码生成的影响。
- 不同技巧显著影响代码正确率和质量，但组合使用未必提升效果。
- 提示技巧存在正确性与质量的权衡，适合优化代码生成的研究者参考。
大型语言模型（LLMs）在软件开发中越来越多地用于代码生成，但其生成内容常出现无关或错误的问题，凸显了提示编程（prompt programming）的重要性。通过应用特定提示技术（如思维链、输入输出示例等），工程师可提升生成代码的质量。尽管已有部分技术被研究，但不同技术及其相互作用对代码生成的影响仍不清晰。本研究提出CodePromptEval，一个包含7072个提示的数据集，用于评估五种提示技术（少样本、角色设定、思维链、函数签名、包列表）在三个模型（GPT-4o、Llama3、Mistral）上生成完整函数时的正确性、相似性和质量表现。结果表明，某些提示技术显著影响生成结果，但组合使用并不一定带来改进；同时观察到正确性与质量之间存在权衡。该数据集及复现包为未来改进和评估提示技术提供了基础。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used by software engineers for code generation. However, limitations of LLMs such as irrelevant or incorrect code have highlighted the need for prompt programming (or prompt engineering) where engineers apply specific prompt techniques (e.g., chain-of-thought or input-output examples) to improve the generated code. While some prompt techniques have been studied, the impact of different techniques -- and their interactions -- on code generation is still not fully understood. In this study, we introduce CodePromptEval, a dataset of 7072 prompts designed to evaluate five prompt techniques (few-shot, persona, chain-of-thought, function signature, list of packages) and their effect on the correctness, similarity, and quality of complete functions generated by three LLMs (GPT-4o, Llama3, and Mistral). Our findings show that while certain prompt techniques significantly influence the generated code, combining multiple techniques does not necessarily improve the outcome. Additionally, we observed a trade-off between correctness and quality when using prompt techniques. Our dataset and replication package enable future research on improving LLM-generated code and evaluating new prompt techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。