用漏洞库评测大模型选安全代码的能力，发现表现参差不齐。
TOSSS: a CVE-based Software Security Benchmark for Large Language Models
- 基于CVE数据库构建双选项代码安全选择任务
- 14个模型在C/C++和Java上得分0.48至0.89
- 可动态扩展新漏洞，适合评估模型安全能力
随着大语言模型（LLMs）能力提升，其被广泛应用于软件开发流程。但它们在安全方面表现如何？我们提出TOSSS（两选项安全代码片段选择），一个基于CVE数据库的基准测试，用于衡量LLMs在安全与漏洞代码间的选择能力。现有安全基准覆盖范围有限，而TOSSS支持随新漏洞披露动态扩展。每个模型获得0到1的得分：1表示始终选安全代码，0表示始终选漏洞代码。我们在C/C++和Java上评估了14个主流开源与闭源模型，得分范围为0.48至0.89。该基准可作为模型性能报告中的补充安全指标。
原文摘要 · Abstract (English)
With their increasing capabilities, Large Language Models (LLMs) are now used across many industries. They have become useful tools for software engineers and support a wide range of development tasks. As LLMs are increasingly used in software development workflows, a critical question arises: are LLMs good at software security? At the same time, organizations worldwide invest heavily in cybersecurity to reduce exposure to disruptive attacks. The integration of LLMs into software engineering workflows may introduce new vulnerabilities and weaken existing security efforts. We introduce TOSSS (Two-Option Secure Snippet Selection), a benchmark that measures the ability of LLMs to choose between secure and vulnerable code snippets. Existing security benchmarks for LLMs cover only a limited range of vulnerabilities. In contrast, TOSSS relies on the CVE database and provides an extensible framework that can integrate newly disclosed vulnerabilities over time. Our benchmark gives each model a security score between 0 and 1 based on its behavior; a score of 1 indicates that the model always selects the secure snippet, while a score of 0 indicates that it always selects the vulnerable one. We evaluate 14 widely used open-source and closed-source models on C/C++ and Java code and observe scores ranging from 0.48 to 0.89. LLM providers already publish many benchmark scores for their models, and TOSSS could become a complementary security-focused score to include in these reports.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。