评估代码漏洞检测模型的行级定位能力,发现其常受非漏洞行干扰。
Evaluating Line-level Localization Ability of Learning-based Code Vulnerability Detection Models
- 提出可解释性评估方法DA,量化预测影响行与真实漏洞行的一致性。
- 多模型测试显示,漏洞预测普遍受非漏洞行干扰,定位偏差严重。
- 适用于各类学习型漏洞检测模型,帮助开发者理解模型决策依据。
为应对软件漏洞这一严峻问题,系统安全常依赖机器学习(ML)算法。然而,现有模型受限于设计,仅能将整个函数标记为漏洞,无法精确定位具体出问题的代码行。而检测粒度对开发人员调试、评估和修复漏洞至关重要。近期研究尝试提升定位能力,缩小检测范围,提供更细粒度的预测。但这些方法往往忽视数据中广泛存在的虚假相关性和偏差,而这些因素通常主导了模型表现。为此,本文提出一种基于可解释性的评估方法——检测一致性(Detection Alignment, DA),量化影响预测的代码行与真实漏洞位置之间的吻合程度。该方法不依赖具体模型,可推广至多种检测任务。通过在多个学习型漏洞检测器和数据集上的分析,我们发现模型预测始终受非漏洞行显著干扰,凸显偏差与虚假相关性的严重影响。代码已开源:https://github.com/pralab/vuln-localization-eval。
原文摘要 · Abstract (English)
To address the extremely concerning problem of software vulnerability, system security is often entrusted to Machine Learning (ML) algorithms. Despite their now established detection capabilities, such models are limited by design to flagging the entire input source code function as vulnerable, rather than precisely localizing the concerned code lines. However, the detection granularity is crucial to support human operators during software development, ensuring that such predictions reflect the true code semantics to help debug, evaluate, and fix the detected vulnerabilities. To address this issue, recent work made progress toward improving the detector's localization ability, thus narrowing down the vulnerability detection "window" and providing more fine-grained predictions. Such approaches, however, implicitly disregard the presence of spurious correlations and biases in the data, which often predominantly influence the performance of ML algorithms. In this work, we investigate how detectors comply with this requirement by proposing an explainability-based evaluation procedure. Our approach, defined as Detection Alignment (DA), quantifies the agreement between the input source code lines that most influence the prediction and the actual localization of the vulnerability as per the ground truth. Through DA, which is model-agnostic and adaptable to different detection tasks, not limited to our use case, we analyze multiple learning-based vulnerability detectors and datasets. As a result, we show how the predictions of such models are consistently biased by non-vulnerable lines, ultimately highlighting the high impact of biases and spurious correlations. The code is available at https://github.com/pralab/vuln-localization-eval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。