arXiv:2510.27675cs.SEcs.CR2025-10被引 1

精心挑选少样本可提升大模型漏洞检测效果,但对不同语言效果差异大。

On the Difficulty of Selecting Few-Shot Examples for Effective LLM-based Vulnerability Detection

  • 按模型常错样本选例,暴露系统性弱点
  • 相似代码检索能提升Python与JS检测准确率
  • C/C++需微调等更复杂方法,少样本无效

大语言模型(LLMs)在代码任务中表现卓越,但在漏洞检测上仍具挑战。上下文学习(ICL)通过在提示中加入少量标注样本来提升性能,其效果高度依赖少样本的选择。本文研究两种直观选择策略:一是基于模型行为,优先选择模型反复出错的样本,以暴露并纠正系统性缺陷;二是基于语义相似度,使用k近邻检索与查询程序相关的上下文。我们在多个编程语言的开源数据集和LLM上进行广泛评估。结果表明,对Python和JavaScript,合理选择少样本可显著提升漏洞检测效果;而对C和C++,少样本选择影响有限,暗示需采用微调或重训练等更昂贵但更有效的方法才能显著提升性能。

原文摘要 · Abstract (English)

Large language models (LLMs) have demonstrated impressive capabilities across a wide range of coding tasks, including summarization, translation, completion, and code generation. Despite these advances, detecting code vulnerabilities remains a challenging problem for LLMs. In-context learning (ICL) has emerged as an effective mechanism for improving model performance by providing a small number of labeled examples within the prompt. Prior work has shown, however, that the effectiveness of ICL depends critically on how these few-shot examples are selected. In this paper, we study two intuitive criteria for selecting few-shot examples for ICL in the context of code vulnerability detection. The first criterion leverages model behavior by prioritizing samples on which the LLM consistently makes mistakes, motivated by the intuition that such samples can expose and correct systematic model weaknesses. The second criterion selects examples based on semantic similarity to the query program, using k-nearest-neighbor retrieval to identify relevant contexts. We conduct extensive evaluations using open-source LLMs and datasets spanning multiple programming languages. Our results show that for Python and JavaScript, careful selection of few-shot examples can lead to measurable performance improvements in vulnerability detection. In contrast, for C and C++ programs, few-shot example selection has limited impact, suggesting that more powerful but also more expensive approaches, such as re-training or fine-tuning, may be required to substantially improve model performance.

漏洞检测少样本学习大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。