arXiv:2505.24451cs.CRcs.AI2025-05被引 8

用线性探针提前评估压缩大模型的漏洞检测效果,省时省力。

LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs

  • 用线性探针在压缩前预测模型性能,避免盲目调参。
  • 可删减33.3%~72.2%层且不损失精度,压缩后准确率达86.9%。
  • 适合需高效部署的网络安全场景,尤其关注推理速度与模型大小。

大型语言模型(LLMs)被广泛用于代码漏洞检测。为提升效率与效果,压缩与微调技术不断发展,但均需大量计算开销。本文分析线性探针(LPs)在压缩前早期估计压缩后模型性能的可行性,并用于确定层剪枝的截止点。提出的方法LPASS应用于BERT和Gemma,在48万条C/C++样本上检测MITRE Top 25中最危险的12类漏洞。线性探针仅需142.97秒即可计算,关键发现:(1) 可分别移除33.3%和72.2%的网络层而无精度损失;(2) 提供微调与压缩后模型性能的早期估计,最低与平均精度误差分别为3%和8.68%。基于LPASS的压缩模型优于现有方法,多类漏洞检测准确率达86.9%。有趣的是,压缩版Gemma在最大F1-score上比原模型高1.6%,同时节省29.4%训练时间、23.8%推理时间和42.98%模型尺寸。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are being extensively used for cybersecurity purposes. One of them is the detection of vulnerable codes. For the sake of efficiency and effectiveness, compression and fine-tuning techniques are being developed, respectively. However, they involve spending substantial computational efforts. In this vein, we analyse how Linear Probes (LPs) can be used to provide an estimation on the performance of a compressed LLM at an early phase -- before fine-tuning. We also show their suitability to set the cut-off point when applying layer pruning compression. Our approach, dubbed $LPASS$, is applied in BERT and Gemma for the detection of 12 of MITRE's Top 25 most dangerous vulnerabilities on 480k C/C++ samples. LPs can be computed in 142.97 s. and provide key findings: (1) 33.3 \% and 72.2\% of layers can be removed, respectively, with no precision loss; (2) they provide an early estimate of the post-fine-tuning and post-compression model effectiveness, with 3\% and 8.68\% as the lowest and average precision errors, respectively. $LPASS$-based LLMs outperform the state of the art, reaching 86.9\% of accuracy in multi-class vulnerability detection. Interestingly, $LPASS$-based compressed versions of Gemma outperform the original ones by 1.6\% of F1-score at a maximum while saving 29.4 \% and 23.8\% of training and inference time and 42.98\% of model size.

漏洞检测模型压缩线性探针LLM安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。