针对Rust漏洞分析，构建可编译且防数据污染的基准测试框架
RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities
- 提供可编译的代码片段，支持多层级漏洞标注
- 模型在漏洞定位上准确率仅约20%，对抗性提示使其下降27%
- 适合评估AI安全分析工具的真实鲁棒性与泛化能力
大型语言模型代理正被广泛用于漏洞分析，但现有基准测试未能跟上发展。它们通常依赖小规模不可编译的代码片段，仅关注二分类（有漏洞或无），且未考虑公开数据集可能已被模型训练所用的风险。本文提出RustMizan，一个面向Rust漏洞分析的基准测试框架，解决了上述问题。该框架包含可在crate、文件和函数级别使用的可编译代码变体，并带有二分类漏洞检测、CWE分类及函数与行级定位标注。通过成对变异机制生成语义保持的代码变异体，用于污染测试与鲁棒性探测。在四种前沿模型的代理设置下，使用命令行接口进行测试，二分类准确率在56%-65%之间，而行级定位F1值约为20%，对抗性提示使行级定位F1下降约27%。
原文摘要 · Abstract (English)
LLM agents are increasingly applied to vulnerability analysis, but existing benchmarks have not kept pace. They typically rely on small non-compilable snippets, focus on binary classification (vulnerable or not), and do not account for the risk that publicly-released datasets are part of model training corpora. We introduce RustMizan, a benchmarking framework for Rust vulnerability analysis that addresses these gaps. RustMizan contains compilable code variants at the crate, file, and function levels, with annotations for binary vulnerability detection, CWE classification, and function- and line-level localization. A paired mutation framework produces semantics-preserving code mutants for contamination testing and robustness probing. Across four frontier models in an agentic setup with command-line access, binary classification sits in the 56-65% range, but line localization F1 stays near 20%, and adversarial cues drop line F1 by about 27%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。