大规模测试发现,大模型修复代码易引入新漏洞,自主性越高风险越大。
How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench
- 用2万+真实GitHub问题测试大模型和智能体修复代码的安全性
- 大模型生成的补丁引入大量新漏洞,且模式与人工代码不同
- 问题描述不完整时,生成漏洞概率显著上升,适合开发者与安全研究者参考
大型语言模型(LLMs)及其智能体框架正被广泛用于自动化程序修复(APR)。尽管已有研究指出大模型生成代码存在安全风险,但多数工作集中在简化或孤立任务上,缺乏真实场景复杂性。本研究首次基于2万多个真实GitHub问题,对开发者、独立大模型(Llama 3.3 Instruct-70B)及三种顶尖智能体框架(OpenHands、AutoCodeRover、HoneyComb)生成的补丁进行大规模安全分析。研究还探讨了代码、问题与项目层面的多种因素,揭示生成不安全补丁的条件。结果表明,Llama引入大量新漏洞,且表现出开发者代码中不存在的独特模式;智能体流程在赋予更高自主权时也产生较多漏洞。漏洞与特定代码特征相关,并常见于缺少关键信息的问题中。这些发现表明上下文因素对补丁安全性至关重要,亟需结合问题与代码层面信息的主动风险评估方法。
原文摘要 · Abstract (English)
Large language models (LLMs) and their agentic frameworks are increasingly adopted to perform development tasks such as automated program repair (APR). While prior work has identified security risks in LLM-generated code, most have focused on synthetic, simplified, or isolated tasks that lack the complexity of real-world program repair. In this study, we present the first large-scale security analysis of LLM-generated patches using 20,000+ GitHub issues. We evaluate patches proposed by developers, a standalone LLM (Llama 3.3 Instruct-70B), and three top-performing agentic frameworks (OpenHands, AutoCodeRover, HoneyComb). Finally, we analyze a wide range of code, issue, and project-level factors to understand the conditions under which generating insecure patches is more likely. Our findings reveal that Llama introduces many new vulnerabilities, exhibiting unique patterns not found in developers' code. Agentic workflows also generate a number of vulnerabilities, particularly when given more autonomy. We find that vulnerabilities in LLM-generated patches are associated with distinctive code characteristics and are commonly observed in issues missing specific types of information. These results suggest that contextual factors play a critical role in the security of the generated patches and point toward the need for proactive risk assessment methods that account for both issue and code-level information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。