用示范学习提升大模型代码生成准确率
Demo-Craft: Using In-Context Learning to Improve Code Generation in Large Language Models
- 通过示范选择和可训练概念嵌入增强上下文理解
- 在MBPP和Humaneval上实现近2倍的通过率提升
- 提出新评估指标，正确性和相似性提升近3倍
利用大语言模型从自然语言指令生成可执行代码面临语义模糊和任务上下文理解困难等问题。为此，我们提出Demo-Craft系统，通过引入示范学习与示范选择，并结合潜在概念学习来增强代码生成能力。潜在概念学习引入可训练的概念标记，捕捉特定任务知识。我们在MBPP和Humaneval两个主流数据集上测试该系统，实验结果表明，相比基线模型，该系统在pass@k指标上提升约2倍。此外，我们提出两个新评估指标：correctness@k和similarity@k，实证研究显示，系统在这两项指标上也实现了近3倍的提升。
原文摘要 · Abstract (English)
Generating executable code from natural language instructions using Large Language Models (LLMs) poses challenges such as semantic ambiguity and understanding taskspecific contexts. To address these issues, we propose a system called DemoCraft, which enhances code generation by leveraging in-context learning and demonstration selection, combined with latent concept learning. Latent concept learning introduces additional concept tokens, which are trainable embeddings that capture task-specific knowledge. We then test our system on two major datasets: MBPP and Humaneval. Our experimental results demonstrate that the proposed system achieves an approximate 2x increase in the pass@k metric compared to baseline models. Furthermore, we introduce two novel evaluation metrics: correctness@k and similarity@k. Our empirical studies indicate that our system attains nearly a 3x improvement in these metrics as well.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。