构建仓库级代码安全评估基准,真实检验AI生成代码的漏洞风险。
A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
- 以真实仓库场景设计评估框架,贴近实际编程任务
- 大模型在复杂项目中仍易生成不安全代码,漏洞率高
- 更大推理预算未必提升安全性,适合关注代码安全的开发者
大型语言模型(LLMs)在软件工程中的广泛应用亟需对其生成代码的安全性进行严格评估。然而,现有基准往往与真实世界中AI辅助编程场景脱节,难以有效衡量生产环境中AI生成代码的实际安全风险。为此,我们提出A.S.E(AI代码生成安全评估),一个面向仓库级别的评估基准,旨在紧密模拟真实世界的AI编程任务,提供全面可靠的评估框架。我们在A.S.E上对主流LLMs的评估揭示了若干关键发现:当前大模型在安全编码方面仍存在显著不足;在仓库级复杂场景下,其表现远逊于片段级任务;更大的推理预算并不必然带来更优的代码生成结果。这些发现揭示了当前AI代码生成的真实状态,帮助开发者识别适用于实际任务的模型,并为改进大模型生成安全高效代码奠定了基础。
原文摘要 · Abstract (English)
The increasing adoption of large language models (LLMs) in software engineering necessitates rigorous security evaluation of their generated code. However, existing benchmarks often lack relevance to real-world AI-assisted programming scenarios, making them inadequate for assessing the practical security risks associated with AI-generated code in production environments. To address this gap, we introduce A.S.E (AI Code Generation Security Evaluation), a repository-level evaluation benchmark designed to closely mirror real-world AI programming tasks, offering a comprehensive and reliable framework for assessing the security of AI-generated code. Our evaluation of leading LLMs on A.S.E reveals several key findings. In particular, current LLMs still struggle with secure coding. The complexity in repository-level scenarios presents challenges for LLMs that typically perform well on snippet-level tasks. Moreover, a larger reasoning budget does not necessarily lead to better code generation. These observations offer valuable insights into the current state of AI code generation and help developers identify the most suitable models for practical tasks. They also lay the groundwork for refining LLMs to generate secure and efficient code in real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。