用小模型生成提示词，让大模型写出更贴近开发者习惯的代码摘要。
Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development
- 用特定任务的Transformer生成提示词，引导大模型生成摘要。
- 实验显示BLEU-4和BLEU-5指标提升5%至7.8%。
- 适合关注代码可读性与安全维护的软件开发人员。
神经源码摘要（NSCS）旨在生成自然语言摘要以提升开发者对代码的理解。在安全软件开发生命周期（SSDLC）的维护阶段，代码摘要能帮助维护者更好理解代码，减少漏洞和错误。然而，许多软件系统的摘要缺失、不完整或过时。现有方案使用小型专用Transformer模型或代码感知的大语言模型（LLM）。专用Transformer生成的摘要在多项自然语言生成（NLG）指标上表现良好，但这些指标更看重词汇重合度而非实际质量。而大语言模型具备捕捉语义的能力，能生成高质量摘要，这得益于近年来大模型的普及和工作站硬件的进步，使得部分大模型可部署在开发者本地。然而，由于其抽象特性，大模型生成的摘要常与开发者书写风格差异较大，导致在NLG指标上得分低。本文提出将两者结合：利用专用Transformer生成提示词，辅助大模型生成摘要。实验中，采用四种不同提示方式，通过四个大模型验证该方法，结果表明在BLEU-4和BLEU-5指标上分别提升5%和7.8%。
原文摘要 · Abstract (English)
Neural Source Code Summarisation (NSCS) aims to generate natural language summaries of source code to improve developers' and maintainers' understanding of code. Source code summaries are vital during the maintenance phase of the Secure Software Development Lifecycle (SSDLC), as they improve maintainers' understanding of code and help reduce the number of bugs and vulnerabilities in a software system. However, summaries are often missing, incomplete, or outdated in many software systems. Solutions to this problem use small, task-specific Transformer models or code-aware Large Language Models (LLMs). Task-specific Transformer-generated summaries often score well across many natural language generation (NLG) metrics, but these metrics reward lexical overlap rather than summary quality. Conversely, the ability of LLMs to capture semantics and produce high-quality summaries presents an exciting solution to this problem. This is especially relevant given the increased availability of LLMs and improvements in workstation hardware in recent years, which mean that some LLMs can now be run on developers' workstations. However, because of their abstractive nature, LLM-generated code summaries often differ greatly from developer-written summaries in the words and phrases they use, resulting in low scores across NLG metrics. We show how combining these two methods, by using Transformer-generated summaries in prompt engineering, may enable LLMs to create better source code summaries and help software practitioners maintain secure systems. We prompt four LLMs using four different prompts, with a task-specific Transformer used to assist the LLMs within the prompts. We present "Transformer-Assisted LLM-Based Source Code Summarisation", a method through which we observe an improvement of 7.8% in BLEU-4 and 5%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。