用真实漏洞数据评估大模型修漏洞能力，发现其常分不清漏洞和修复代码。
VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patching
- 基于307个真实Linux内核漏洞构建评测框架，含原始与修复代码
- 大模型在识别漏洞时准确率不高，修复代码常过于简化无法直接使用
- 适合关注代码安全自动化研究的开发者和安全研究人员
大型语言模型（LLMs）在代码翻译等任务中展现出潜力，引发了对其在软件漏洞检测（SVD）和补丁生成（SVP）中应用的兴趣。为推动该领域研究，建立基准测试至关重要。本文提出VulnLLMEval框架，用于评估LLMs在识别和修补C代码漏洞方面的能力。研究基于307个来自Linux内核的真实漏洞，构建了一个包含脆弱代码与修复代码的高质量数据集。该数据集覆盖多样且具代表性，为评估模型在SVD和SVP任务中的表现提供了可靠基础。实验结果表明，LLMs在区分漏洞代码与修复代码方面存在困难；在生成补丁时，模型倾向于过度简化，产生的解决方案通常需进一步人工修正才能使用。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown promise in tasks like code translation, prompting interest in their potential for automating software vulnerability detection (SVD) and patching (SVP). To further research in this area, establishing a benchmark is essential for evaluating the strengths and limitations of LLMs in these tasks. Despite their capabilities, questions remain regarding whether LLMs can accurately analyze complex vulnerabilities and generate appropriate patches. This paper introduces VulnLLMEval, a framework designed to assess the performance of LLMs in identifying and patching vulnerabilities in C code. Our study includes 307 real-world vulnerabilities extracted from the Linux kernel, creating a well-curated dataset that includes both vulnerable and patched code. This dataset, based on real-world code, provides a diverse and representative testbed for evaluating LLM performance in SVD and SVP tasks, offering a robust foundation for rigorous assessment. Our results reveal that LLMs often struggle with distinguishing between vulnerable and patched code. Furthermore, in SVP tasks, these models tend to oversimplify the code, producing solutions that may not be directly usable without further refinement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。