用算法计划检索代码示例,提升冷门语言生成效果
PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation
- 用伪代码提取算法计划作为查询,精准匹配有用示例
- 在多个评测集上优于当前最优方法,跨语言生成也更稳定
- 适合冷门编程语言或少样本场景的代码生成任务
大型语言模型在代码生成中展现巨大潜力,尤其结合少样本示例的检索增强生成(RAG)时。然而,如何选取能有效提升生成质量的示例仍具挑战,特别是在目标编程语言(PL)数据稀缺时。本研究发现:(1) 检索那些包含可复用算法计划的示例能显著提高生成准确率;(2) 将代码转为伪代码可有效捕捉算法计划,即使源语言与目标语言不同也能提升检索质量。基于此,我们提出针对少样本提示的“计划即查询”示例检索框架(PERC),利用算法计划识别并检索高效示例。在CodeContests、HumanEval和MultiPL-E等多个基准上进行大量实验验证:无论源语言与目标语言是否一致,PERC均持续优于现有最先进RAG方法,展现出在多样化编码环境中的适应性与鲁棒性。
原文摘要 · Abstract (English)
Code generation with large language models has shown significant promise, especially when employing retrieval-augmented generation (RAG) with few-shot examples. However, selecting effective examples that enhance generation quality remains a challenging task, particularly when the target programming language (PL) is underrepresented. In this study, we present two key findings: (1) retrieving examples whose presented algorithmic plans can be referenced for generating the desired behavior significantly improves generation accuracy, and (2) converting code into pseudocode effectively captures such algorithmic plans, enhancing retrieval quality even when the source and the target PLs are different. Based on these findings, we propose Plan-as-query Example Retrieval for few-shot prompting in Code generation (PERC), a novel framework that utilizes algorithmic plans to identify and retrieve effective examples. We validate the effectiveness of PERC through extensive experiments on the CodeContests, HumanEval and MultiPL-E benchmarks: PERC consistently outperforms the state-of-the-art RAG methods in code generation, both when the source and target programming languages match or differ, highlighting its adaptability and robustness in diverse coding environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。