用提示词工程提升大模型修复漏洞能力，实测效果显著
Identifying Helpful Context for LLM-based Vulnerability Repair: A Preliminary Study
- 设计九种含不同上下文的提示词，测试其对漏洞修复的影响
- 结合CVE信息与代码上下文，修复成功率提升至62%（26个漏洞）
- 适合安全研究者和自动化修复工具开发者参考
近期大型语言模型在软件漏洞检测与修复方面展现出潜力。本文研究GPT-4o在修复Vul4J数据集中的Java漏洞时的表现，探究不同上下文信息对自动化漏洞修复（AVR）能力的影响。将GPT-4o与先前使用GPT-4的结果在相同提示下进行对比，评估了我们设计的九种含CWE、CVE信息及手动提取代码上下文的提示词。每个提示在42个漏洞上运行三次，修复候选结果通过Vul4J自动化测试框架验证。结果显示，GPT-4o平均性能比GPT-4低11.9%，但三轮总修复率高出10.5%。CVE信息显著提升修复率，任务描述长度影响较小。结合CVE引导与手动代码上下文的提示表现最佳。使用我们提出的 extsc{Top}-3提示组合，GPT-4o至少一次修复了26个漏洞（占总数62%），优于原始基线（40%）和复现结果（45%），表明集成提示策略可在零样本场景下有效提升漏洞修复效果。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have shown promise for automated vulnerability detection and repair in software systems. This paper investigates the performance of GPT-4o in repairing Java vulnerabilities from a widely used dataset (Vul4J), exploring how different contextual information affects automated vulnerability repair (AVR) capabilities. We compare the latest GPT-4o's performance against previous results with GPT-4 using identical prompts. We evaluated nine additional prompts crafted by us that contain various contextual information such as CWE or CVE information, and manually extracted code contexts. Each prompt was executed three times on 42 vulnerabilities, and the resulting fix candidates were validated using Vul4J's automated testing framework. Our results show that GPT-4o performed 11.9\% worse on average than GPT-4 with the same prompt, but was able to fix 10.5\% more distinct vulnerabilities in the three runs together. CVE information significantly improved repair rates, while the length of the task description had minimal impact. Combining CVE guidance with manually extracted code context resulted in the best performance. Using our \textsc{Top}-3 prompts together, GPT-4o repaired 26 (62\%) vulnerabilities at least once, outperforming both the original baseline (40\%) and its reproduction (45\%), suggesting that ensemble prompt strategies could improve vulnerability repair in zero-shot settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。