分析319个代码修复补丁，发现大模型修复安全漏洞效果不佳。
Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation
- 用三轴评估法检验补丁正确性
- 仅24.8%补丁完全正确，51.4%同时不安全也不可用
- 补丁常因理解错误导致修复失败，适合安全研究者参考
大型语言模型（LLMs）在自动化程序修复（APR）中展现潜力，但其在安全漏洞修复中的有效性尚未明确。本研究分析了基于Vul4J基准的64个Java漏洞生成的319个LLM补丁。通过三轴评估（编译、利用验证测试验证安全性、测试套件验证功能），结果表明仅有24.8%的补丁达到完全正确，51.4%同时在安全性和功能性上失败。主要失败模式为语义误解：模型生成语法正确但修复策略错误的代码。提出的安全修复评分（SRS）量化该差距，显示模型平均保留功能（0.832），但安全性严重不足（0.251）。漏洞类型显著影响修复难度，修复率从0%（输入验证）到45%（无限循环）不等。研究强调，部署前必须对LLM生成的安全补丁进行严格验证。
原文摘要 · Abstract (English)
Large Language Models (LLMs) show promise for Automated Program Repair (APR), yet their effectiveness on security vulnerabilities remains poorly characterized. This study analyzes 319 LLM-generated security patchesacross 64 Java vulnerabilities from the Vul4J benchmark. Using tri-axis evaluation (compilation, security via PoV tests, functionality via test suites), the analysis reveals that only 24.8% of patches achieve full correctness, while 51.4% fail both security and functionality. The dominant failure mode is semantic misunderstanding: LLMs produce syntactically valid code but apply incorrect repair strategies. The proposed Security Repair Score (SRS) quantifies this gap, showing LLMs preserve functionality (mean 0.832) but struggle with security (mean 0.251). Vulnerability type strongly predicts difficulty, with fix rates ranging from 0% (input validation) to 45% (infinite loop). These findings demonstrate that LLM security patches require rigorous validation before deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。