新基准测试代码生成安全漏洞，发现顶尖模型仅23.8%正确修复。
SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
- 从真实开源漏洞中提取105个需多文件修改的编码任务
- 用静态动态检测结合验证，最佳模型仅23.8%解出安全正确答案
- 适合研究代码生成安全性的研究人员和工具开发者
大语言模型驱动的代码代理正快速改变软件工程，但其生成代码的安全风险日益突出。现有基准未能涵盖开发者实际引入漏洞的场景，导致人与代理的公平对比难以实现。为此，我们提出SecureVibeBench，包含105个源自OSS-Fuzz中41个项目的C/C++安全编码任务。该基准具有三大特点：(i) 需在大型仓库中进行多文件编辑的真实任务设置；(ii) 基于真实开源漏洞构建，精确定位漏洞引入点；(iii) 结合功能测试与安全检查，采用静态与动态断言进行综合评估。我们在5个主流代码代理（如OpenHands）上评估，使用5个LLM（如Claude sonnet 4.5）。结果表明，当前代理难以同时生成正确且安全的代码，即使表现最好的模型，在SecureVibeBench上也仅能生成23.8%正确且安全的解决方案。代码与数据已公开于https://github.com/iCSawyer/SecureVibeBench。
原文摘要 · Abstract (English)
Large language model-powered code agents are rapidly transforming software engineering, yet the security risks of their generated code have become a critical concern. Existing benchmarks have provided valuable insights, but they fail to capture scenarios in which vulnerabilities are actually introduced by human developers, making fair comparisons between humans and agents infeasible. We therefore introduce SecureVibeBench, a benchmark of 105 C/C++ secure coding tasks sourced from 41 projects in OSS-Fuzz for code agents. SecureVibeBench has the following features: (i) realistic task settings that require multi-file edits in large repositories, (ii)~aligned contexts based on real-world open-source vulnerabilities with precisely identified vulnerability introduction points, and (iii) comprehensive evaluation that combines functionality testing and security checking with both static and dynamic oracles. We evaluate 5 popular code agents like OpenHands, supported by 5 LLMs (e.g., Claude sonnet 4.5) on SecureVibeBench. Results show that current agents struggle to produce both correct and secure code, as even the best-performing one, produces merely 23.8\% correct and secure solutions on SecureVibeBench. Our code and data are on https://github.com/iCSawyer/SecureVibeBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。