用检索增强提示提升代码漏洞检测效果,无需微调模型。
Retrieval-Augmented Few-Shot Prompting Versus Fine-Tuning for Code Vulnerability Detection
- 从语义相似代码中检索示例,动态增强提示
- 20个示例下F1达74.05%,部分匹配准确率83.90%
- 比零样本和微调模型更高效,适合资源有限场景
少样本提示已成为利用大语言模型在专业任务中能力的实用替代方案。然而其效果高度依赖于上下文示例的选择与质量,尤其在复杂领域。本文研究检索增强提示策略,以提升代码漏洞检测中的少样本表现。目标是从预定义漏洞类别中识别给定代码片段中的一个或多个安全弱点。我们使用Gemini-1.5-Flash模型对三种方法进行系统评估:(1) 随机选择示例的标准少样本提示;(2) 使用语义相似示例的检索增强提示;(3) 基于检索示例的标签分配(无模型推理)。结果表明,检索增强提示始终优于其他提示策略。在20个示例时,取得F1分数74.05%与部分匹配准确率83.90%。进一步对比零样本提示及多个微调模型(包括Gemini-1.5-Flash与较小开源模型DistilBERT、DistilGPT2、CodeBERT),检索增强提示优于零样本(F1: 36.35%,部分匹配准确率: 20.30%)与微调Gemini(F1: 59.31%,部分匹配准确率: 53.10%),且避免了模型微调的训练时间与成本。而微调CodeBERT表现更高(F1: 91.22%,部分匹配准确率: 91.30%),但需额外训练、维护与资源投入。
原文摘要 · Abstract (English)
Few-shot prompting has emerged as a practical alternative to fine-tuning for leveraging the capabilities of large language models (LLMs) in specialized tasks. However, its effectiveness depends heavily on the selection and quality of in-context examples, particularly in complex domains. In this work, we examine retrieval-augmented prompting as a strategy to improve few-shot performance in code vulnerability detection, where the goal is to identify one or more security-relevant weaknesses present in a given code snippet from a predefined set of vulnerability categories. We perform a systematic evaluation using the Gemini-1.5-Flash model across three approaches: (1) standard few-shot prompting with randomly selected examples, (2) retrieval-augmented prompting using semantically similar examples, and (3) retrieval-based labeling, which assigns labels based on retrieved examples without model inference. Our results show that retrieval-augmented prompting consistently outperforms the other prompting strategies. At 20 shots, it achieves an F1 score of 74.05% and a partial match accuracy of 83.90%. We further compare this approach against zero-shot prompting and several fine-tuned models, including Gemini-1.5-Flash and smaller open-source models such as DistilBERT, DistilGPT2, and CodeBERT. Retrieval-augmented prompting outperforms both zero-shot (F1 score: 36.35%, partial match accuracy: 20.30%) and fine-tuned Gemini (F1 score: 59.31%, partial match accuracy: 53.10%), while avoiding the training time and cost associated with model fine-tuning. On the other hand, fine-tuning CodeBERT yields higher performance (F1 score: 91.22%, partial match accuracy: 91.30%) but requires additional training, maintenance effort, and resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。