用Llama模型检测代码漏洞,微调比提示工程更有效。
Llama-based source code vulnerability detection: Prompt engineering vs Fine tuning
- 提出双阶段微调(Double Fine-tuning)提升漏洞检测效果
- 在BigVul和PrimeVul数据集上验证了微调的有效性
- 提示工程效果差,但检索增强生成(RAG)表现较好
过去二十年软件开发周期加速,导致漏洞数量持续上升。自动化源代码漏洞检测(CVD)变得至关重要,已有多种方法从传统程序分析到最新AI技术。本研究评估大型语言模型(LLMs)在CVD任务中的表现,探索不同微调与提示工程策略。特别提出一种名为双微调(Double Fine-tuning)的新方法,并测试了较少研究的测试时微调。基于开源Llama-3.1 8B模型,使用来自BigVul和PrimeVul数据集的代码样本进行实验。结果表明微调对任务解决至关重要,双微调表现优异,且Llama模型在漏洞检测中具有潜力。尽管提示工程效果不佳,但检索增强生成(RAG)作为示例选择技术表现相对良好。部分问题已解答,更多方向有待未来研究。代码仓库:https://github.com/DynaSoumhaneOuchebara/Llama-based-vulnerability-detection。
原文摘要 · Abstract (English)
The significant increase in software production, driven by the acceleration of development cycles over the past two decades, has led to a steady rise in software vulnerabilities, as shown by statistics published yearly by the CVE program. The automation of the source code vulnerability detection (CVD) process has thus become essential, and several methods have been proposed ranging from the well established program analysis techniques to the more recent AI-based methods. Our research investigates Large Language Models (LLMs), which are considered among the most performant AI models to date, for the CVD task. The objective is to study their performance and apply different state-of-the-art techniques to enhance their effectiveness for this task. We explore various fine-tuning and prompt engineering settings. We particularly suggest one novel approach for fine-tuning LLMs which we call Double Fine-tuning, and also test the understudied Test-Time fine-tuning approach. We leverage the recent open-source Llama-3.1 8B, with source code samples extracted from BigVul and PrimeVul datasets. Our conclusions highlight the importance of fine-tuning to resolve the task, the performance of Double tuning, as well as the potential of Llama models for CVD. Though prompting proved ineffective, Retrieval augmented generation (RAG) performed relatively well as an example selection technique. Overall, some of our research questions have been answered, and many are still on hold, which leaves us many future work perspectives. Code repository is available here: https://github.com/DynaSoumhaneOuchebara/Llama-based-vulnerability-detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。