用上下文学习提升代码与文本联合分析能力,显著改进软件工程任务表现。
In-Context Code-Text Learning for Bimodal Software Engineering
- 设计可配置提示模板,统一代码与文本的上下文学习流程。
- 在23个任务中实现至少12%精度提升,程序修复任务BLEU增85点。
- 适合需要高效少样本处理的开发者与研究者使用。
大语言模型使双模态软件分析看似可行,但代码与自然语言之间的复杂交互仍阻碍模型泛化。本文提出针对代码-文本双模态的上下文学习方法,基于预训练CodeLLAMA模型开展系统研究。构建涵盖23项软件工程任务的数据集,并转化为上下文学习格式。提出可配置提示模板以有效提取特征,构建统一的InCTRL流水线。在多个任务上的实证表明,InCTRL模型在少样本场景下超越现有最优模型,包括CodeLLAMA。例如,在程序修复任务中,精度提升至少12%,召回率达93.88%;在克隆检测中,准确率提高69个百分点;程序修复任务的BLEU分数提升85点。此外,结合检索增强生成时,InCTRL在单个下游任务上也达到最先进水平。定性分析揭示其优势,并开源全部模型与数据集以促进广泛影响。
原文摘要 · Abstract (English)
Bimodal software analysis initially appeared to be within reach with the advent of large language models. Unfortunately, the complex interplay of natural language text and code in software engineering, presents unique challenges that prevent pretrained models to generalize to a variety of tasks. We postulate that in-context learning for the code-text bimodality is a promising avenue. This paper thus introduces a comprehensive study of in-context code-text learning, focusing on leveraging pretrained CodeLLAMA models. We consider a diverse dataset encompassing 23 software engineering tasks, which we transform in an in-context learning format. To effectively extract informative features, we propose a configurable prompt template. Our proposed pipeline, InCTRL, then unifies prompt learning across various software engineering tasks. Extensive evaluation on the study datasets demonstrates the superiority of INCTRL-models in few-shot performance, surpassing state-of-the-art models including the support model, CodeLLAMA. Typically, we observe that applied to the CodeLLAMA model, INCTRL brings improvements in terms of precision (at least about 12\%) and recall (up to 93.88\%) on various tasks. For example, on the task of program repair, INCTRL improves the BLEU score of CodeLLAMA by 85 points, while for clone detection, INCTRL achieves an improvement of 69 percentage points. Moreover, INCTRL-models offer state-of-the-art performance when using retrieval-augmented generation on individual downstream tasks. Finally, we qualitatively analyze the benefits of INCTRL over CodeLLAMA and open-source all models for broader impact. We make our code and dataset publicly available at: \begin{center} {\url{https://anonymous.4open.science/r/inctrl-B65B}} \end{center}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。