测试12个代码代理在真实漏洞任务中的安全表现，发现多数生成代码不安全。
Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks
- 构建包含186个真实漏洞任务的基准SUSVIBES，评估代码代理安全性。
- 即使57%的代码功能正确，仅有11.8%的输出无安全漏洞。
- 添加漏洞提示等初步策略也无法有效提升代码安全性，适合关注AI生成代码风险的研究者。
Vibe coding 是一种新型软件开发范式，人类工程师通过少量指令让大语言模型（LLM）代理完成复杂编码任务。尽管该模式日益普及，其生成代码是否安全仍存疑。为此，我们提出SUSVIBES基准，涵盖来自真实开源项目的186个功能需求任务，这些任务中人类程序员曾引入漏洞。我们在该基准上评估了12种主流编码代理设置与前沿模型的表现。令人震惊的是，所有代理在软件安全方面均表现不佳。尽管使用Claude 4 Sonnet的SWE-Agent有57%的解决方案功能正确，但仅11.8%是安全的。进一步实验表明，诸如在需求中加入漏洞提示等初步安全策略无法缓解上述问题。研究结果警示：vibe coding在安全敏感场景中广泛采用需谨慎。代码与数据集已公开于https://github.com/LeiLiLab/susvibes，排行榜见https://leililab.github.io/susvibes-leaderboard。
原文摘要 · Abstract (English)
Vibe coding is a new software development paradigm in which human engineers prompt a large language model (LLM) agent to complete complex coding tasks with little supervision. Although vibe coding is increasingly adopted, is the generated code really safe to deploy in production? To investigate this question, we propose SUSVIBES, a benchmark consisting of 186 feature-request software engineering tasks from real-world open-source projects, for which, human programmers committed vulnerable implementations. We evaluate 12 widely used coding agentic settings with frontier models on the benchmark. Disturbingly, all agents perform poorly in terms of software security. Although 57% of the solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 11.8% are secure. Further experiments demonstrate that preliminary security strategies, such as augmenting the feature request with vulnerability hints, cannot mitigate these security issues. Our findings raise serious concerns about the widespread adoption of vibe coding, particularly in security-sensitive applications. The code and dataset are available at https://github.com/LeiLiLab/susvibes. The leaderboard is at https://leililab.github.io/ susvibes-leaderboard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。