评测大模型生成安全代码能力,覆盖5语言22类漏洞的真实工业场景。
SecCodeBench-V2 Technical Report
- 基于阿里工业真实案例构建函数级任务,明确接口与依赖。
- 98个场景含可执行验证用例,专家双审确保测试可靠性。
- 动态执行+大模型判官双重验证,支持跨模型可比评估。
我们提出SecCodeBench-V2,一个公开的基准,用于评估大型语言模型(LLM)编程助手生成安全代码的能力。该基准包含98个生成与修复场景,源自阿里巴巴集团的工业生产,涵盖五种编程语言(Java、C、Python、Go、JavaScript)中的22类常见弱性(CWE)。每个场景采用函数级任务形式:提供完整项目框架,要求模型在固定接口和依赖下实现或修补目标函数。每个场景均配备可执行的原型验证(PoC)测试用例,用于功能与安全双重验证。所有测试用例均由安全专家编写并双审,确保高保真度、广泛覆盖与可靠真值。除基准本身外,我们构建统一评估流程,主要通过动态执行评估模型表现。多数场景中,在隔离环境中编译运行模型生成物,并执行PoC用例以验证功能正确性与安全属性;对于无法通过确定性测试判定的安全问题,额外引入大模型作为裁判。为综合评估异构场景与难度差异,设计基于Pass@K的评分协议,对场景与严重性进行合理聚合,实现跨模型的全面可比评估。整体上,SecCodeBench-V2为评估AI编程助手中的安全能力提供了严谨且可复现的基础,相关结果与资源已发布于https://alibaba.github.io/sec-code-bench,基准代码开放于https://github.com/alibaba/sec-code-bench。
原文摘要 · Abstract (English)
We introduce SecCodeBench-V2, a publicly released benchmark for evaluating Large Language Model (LLM) copilots' capabilities of generating secure code. SecCodeBench-V2 comprises 98 generation and fix scenarios derived from Alibaba Group's industrial productions, where the underlying security issues span 22 common CWE (Common Weakness Enumeration) categories across five programming languages: Java, C, Python, Go, and JavaScript. SecCodeBench-V2 adopts a function-level task formulation: each scenario provides a complete project scaffold and requires the model to implement or patch a designated target function under fixed interfaces and dependencies. For each scenario, SecCodeBench-V2 provides executable proof-of-concept (PoC) test cases for both functional validation and security verification. All test cases are authored and double-reviewed by security experts, ensuring high fidelity, broad coverage, and reliable ground truth. Beyond the benchmark itself, we build a unified evaluation pipeline that assesses models primarily via dynamic execution. For most scenarios, we compile and run model-generated artifacts in isolated environments and execute PoC test cases to validate both functional correctness and security properties. For scenarios where security issues cannot be adjudicated with deterministic test cases, we additionally employ an LLM-as-a-judge oracle. To summarize performance across heterogeneous scenarios and difficulty levels, we design a Pass@K-based scoring protocol with principled aggregation over scenarios and severity, enabling holistic and comparable evaluation across models. Overall, SecCodeBench-V2 provides a rigorous and reproducible foundation for assessing the security posture of AI coding assistants, with results and artifacts released at https://alibaba.github.io/sec-code-bench. The benchmark is publicly available at https://github.com/alibaba/sec-code-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。