研究少样本示例如何提升代码生成大模型表现
Does Few-Shot Learning Help LLM Performance in Code Synthesis?
- 提出两种选示例方法：无需训练的CODEEXEMPLAR-FREE和依赖模型的CODEEXEMPLAR-BASED
- 在HumanEval+上显著提升CodeLlama的代码生成能力
- 为提示工程中少样本示例选择提供可解释的实用方案
大型语言模型（LLMs）在代码生成方面已取得显著进展，主要得益于模型设计、训练策略及思维链方法的改进。然而，提示层面的优化仍是重要但未被充分探索的方向。本文聚焦于多数代码生成提示中存在的少样本示例，系统研究其是否能提升LLM的编码能力，哪些示例影响最大，以及如何选择有效示例。本文提出两种示例选择方法：一种不依赖模型训练的CODEEXEMPLAR-FREE，另一种基于模型的CODEEXEMPLAR-BASED，二者在性能提升与训练数据依赖性、可解释性之间形成权衡。两种方法均在主流的HumanEval+基准上显著提升了CodeLlama的代码生成能力。本研究为代码生成提示中少样本示例的选择提供了有价值的实践指导。
原文摘要 · Abstract (English)
Large language models (LLMs) have made significant strides at code generation through improved model design, training, and chain-of-thought. However, prompt-level optimizations remain an important yet under-explored aspect of LLMs for coding. This work focuses on the few-shot examples present in most code generation prompts, offering a systematic study on whether few-shot examples improve LLM's coding capabilities, which few-shot examples have the largest impact, and how to select impactful examples. Our work offers 2 approaches for selecting few-shot examples, a model-free method, CODEEXEMPLAR-FREE, and a model-based method, CODEEXEMPLAR-BASED. The 2 methods offer a trade-off between improved performance and reliance on training data and interpretability. Both methods significantly improve CodeLlama's coding ability across the popular HumanEval+ coding benchmark. In summary, our work provides valuable insights into how to pick few-shot examples in code generation prompts to improve LLM code generation capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。