arXiv:2602.03012cs.CRcs.AI2026-02中稿 · ICML被引 4

自动构建高保真代码安全漏洞任务,提升模型评估与训练效率

CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability

  • 用多智能体框架将零散漏洞元数据转为可执行任务
  • 95%正确率、96%环境保真度,验证效果达专家水平
  • 生成超1000个训练环境,支持持续更新的实时漏洞评测

评估和提升代码智能体的安全能力需要高质量、可执行的漏洞任务。然而,现有方法依赖昂贵且不可扩展的手动复现,且数据分布过时。为此,我们提出CVE-Factory,首个实现专家级质量的多智能体框架,可自动将稀疏的CVE元数据转化为完整可执行的智能体任务。与人工专家复现交叉验证显示,CVE-Factory在解决方案正确性上达到95%,环境保真度达96%,证实其具备专家级质量。在最新真实漏洞上测试,验证成功率提升至66.2%。该自动化带来两项下游贡献:一是构建了LiveCVEBench,一个持续更新的基准,包含190个任务,覆盖14种语言和153个仓库,涵盖包括AI工具链漏洞在内的新兴威胁;二是合成超过1000个可执行训练环境,首次实现代码安全领域智能体任务的大规模扩展。微调后的Qwen3-32B模型在LiveCVEBench上表现从5.3%提升至35.8%,超越Claude 4.5 Sonnet,且在Terminal Bench上也从12.5%提升至31.3%。所有资源已开源,详见https://github.com/livecvebench/CVE-Factory。

原文摘要 · Abstract (English)

Evaluating and improving the security capabilities of code agents requires high-quality, executable vulnerability tasks. However, existing works rely on costly, unscalable manual reproduction and suffer from outdated data distributions. To address these, we present CVE-Factory, the first multi-agent framework to achieve expert-level quality in automatically transforming sparse CVE metadata into fully executable agentic tasks. Cross-validation against human expert reproductions shows that CVE-Factory achieves 95\% solution correctness and 96\% environment fidelity, confirming its expert-level quality. It is also evaluated on the latest realistic vulnerabilities and achieves a 66.2\% verified success. This automation enables two downstream contributions. First, we construct LiveCVEBench, a continuously updated benchmark of 190 tasks spanning 14 languages and 153 repositories that captures emerging threats including AI-tooling vulnerabilities. Second, we synthesize over 1,000 executable training environments, the first large-scale scaling of agentic tasks in code security. Fine-tuned Qwen3-32B improves from 5.3\% to 35.8\% on LiveCVEBench, surpassing Claude 4.5 Sonnet, with gains generalizing to Terminal Bench (12.5\% to 31.3\%). We open-source CVE-Factory, LiveCVEBench, Abacus-cve (fine-tuned model), training dataset, and leaderboard. All resources are available at https://github.com/livecvebench/CVE-Factory .

代码安全智能体漏洞检测自动化评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。