arXiv:2605.13138cs.SEcs.CR2026-05中稿 · ASE26

代码模型难精准识别漏洞修复,真实效果被高估。

A Comprehensive Evaluation of Code Language Models for Security Patch Detection

论文配图:A Comprehensive Evaluation of Code Language Models for Security Patch Detection
图 1 · 摘自论文原文
  • 构建统一框架,控制数据泄露与标签误差,仅用代码判断修复
  • 80%以上漏洞修复被模型遗漏,即使大模型也无显著提升
  • 适合关注安全检测可靠性的研究人员和工程团队

自动化检测漏洞修复提交(VFCs)对及时部署安全补丁至关重要,因公告数据库平均滞后补丁发布25天,且许多修复未生成公告。代码语言模型被广泛用于识别VFCs,但其能否仅凭代码变更本身识别安全修复仍不明确,因现有性能受提交信息、项目级数据泄露及数据质量不确定性影响。本文通过整合20个分散数据集(超18万次提交),建立统一评估框架,训练了270个参数量从125M到800亿的模型,在分组分层的纯代码评估下,隔离代码信号,评估模型容量及上下文(从函数内扩展到跨函数仓库上下文)的影响。结果显示:模型容量虽带来明显但有限的提升,所测试的上下文信号在严格假阳性预算下均无可靠改进;在假阳性率0.5%条件下,所有微调后的纯代码模型至少遗漏80%的漏洞修复。人工专家审计进一步表明,标签错误集中于无CVE关联的提交,严重扭曲评估结果。研究提出针对聚合VFC数据集的评估建议,并公开统一框架与评估套件。

原文摘要 · Abstract (English)

Automated detection of vulnerability-fixing commits (\vfcs) is critical for timely security patch deployment, as advisory databases lag patch releases by a median of 25 days and many fixes never receive advisories. Code language models are increasingly adopted for identifying VFCs, yet whether they can recognize a security fix from the code change itself remains unclear, as reported performance is shaped by commit messages, project-level data leakage, and uncertain data quality. We present a rigorous re-evaluation that jointly controls these factors through a unified framework consolidating 20 fragmented datasets spanning more than 180000 commits. Training 270 models from 125M to 80B parameters, we isolate the code signal under group-stratified code-only evaluation and assess the impact of model capacity and additional code context, ranging from intra-procedural enrichment to inter-procedural repository context. Model capacity yields clear but insufficient gains, and the evaluated context signals provide no reliable improvement under strict false positive budgets. At a false positive rate of 0.5%, every evaluated fine-tuned code-only model misses at least 80% of vulnerability fixes. A manual expert audit further shows that label error concentrates in commits lacking CVE association and primarily distorts evaluation. We derive concrete recommendations for evaluating on aggregated VFC datasets and release our unified framework and evaluation suite.

漏洞检测代码模型安全评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。