评测大模型对真实与人工漏洞的修复效果，发现真实漏洞更易修复。
Evaluating LLMs for One-Shot Patching of Real and Artificial Vulnerabilities
- 用漏洞验证测试评估多个大模型的修复能力
- 真实漏洞修复成功率高于人工漏洞
- 不同模型修复范围差异大，适合组合使用
自动化漏洞修复对软件安全至关重要，近期大型语言模型（LLMs）在该任务上展现出潜力。然而，现有研究主要基于公开披露的真实漏洞，对人工构造漏洞的修复效果仍不明确。本研究实证评估了GPT系列、LLaMA、DeepSeek和Mistral等主流LLMs在真实与人工漏洞上的修复表现。通过漏洞证明（PoV）测试执行，检验生成代码是否真正修复漏洞。结果表明，LLMs在真实漏洞上的修复效果优于人工漏洞。同时，各模型间存在显著的重叠（多模型修复相同漏洞）与互补性（仅单模型能修复特定漏洞），凸显了选择合适模型的重要性。
原文摘要 · Abstract (English)
Automated vulnerability patching is crucial for software security, and recent advancements in Large Language Models (LLMs) present promising capabilities for automating this task. However, existing research has primarily assessed LLMs using publicly disclosed vulnerabilities, leaving their effectiveness on related artificial vulnerabilities largely unexplored. In this study, we empirically evaluate the patching effectiveness and complementarity of several prominent LLMs, such as OpenAI's GPT variants, LLaMA, DeepSeek, and Mistral models, using both real and artificial vulnerabilities. Our evaluation employs Proof-of-Vulnerability (PoV) test execution to concretely assess whether LLM-generated source code successfully patches vulnerabilities. Our results reveal that LLMs patch real vulnerabilities more effectively compared to artificial ones. Additionally, our analysis reveals significant variability across LLMs in terms of overlapping (multiple LLMs patching the same vulnerabilities) and complementarity (vulnerabilities patched exclusively by a single LLM), emphasizing the importance of selecting appropriate LLMs for effective vulnerability patching.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。