用代码复杂度自动选最佳提示工程,提升生成准确率并省 token
Selection of Prompt Engineering Techniques for Code Generation through Predicting Code Complexity
- 以代码复杂度为指标,智能选择最适合的提示工程技术
- 在 MBPP 与 HumanEval 上提升 pass@1 准确率最高达 1.9%,节省 74.8% token
- 无需依赖具体提示方法,适合作为通用代码生成优化框架
大型语言模型在软件工程任务中表现优异,但生成正确可靠代码的准确性仍面临挑战。尽管已有多种提示工程技巧(PETs)被提出,但并无单一方法适用于所有场景。选择合适技巧的难点在于:(1) 交互式提示对简单问题未必有效;(2) 现有自动化方法缺乏适应性,未充分利用多阶段响应。为此,我们提出 PET-Select——一种不依赖特定 PET 的选择模型,通过代码复杂度作为代理指标分类查询,并匹配最适配的 PET。结合对比学习,该模型能有效区分简单与复杂问题,从而为不同复杂度的查询选择最优提示策略。在 MBPP 与 HumanEval 基准上使用 GPT-3.5 Turbo 与 GPT-4o 的评估显示,最高提升 pass@1 准确率 1.9%,同时减少 74.8% 的 token 使用。定量与定性分析进一步验证了其在精准选择提示技术方面的有效性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated impressive performance in software engineering tasks. However, improving their accuracy in generating correct and reliable code remains challenging. Numerous prompt engineering techniques (PETs) have been developed to address this, but no single approach is universally optimal. Selecting the right PET for each query is difficult for two primary reasons: (1) interactive prompting techniques may not consistently deliver the expected benefits, especially for simpler queries, and (2) current automated prompt engineering methods lack adaptability and fail to fully utilize multi-stage responses. To overcome these challenges, we propose PET-Select, a PET-agnostic selection model that uses code complexity as a proxy to classify queries and select the most appropriate PET. By incorporating contrastive learning, PET-Select effectively distinguishes between simple and complex problems, allowing it to choose PETs that are best suited for each query's complexity level. Our evaluations on the MBPP and HumanEval benchmarks using GPT-3.5 Turbo and GPT-4o show up to a 1.9% improvement in pass@1 accuracy, along with a 74.8% reduction in token usage. Additionally, we provide both quantitative and qualitative results to demonstrate how PET-Select effectively selects the most appropriate techniques for each code generation query, further showcasing its efficiency in optimizing PET selection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。