代码度量比大模型更懂漏洞发现，揭示其本质局限
LLM-based Vulnerability Discovery through the Lens of Code Metrics
- 用经典代码度量训练分类器，效果媲美顶尖大模型
- 改变度量值时大模型预测随之同步变化，存在强依赖关系
- 研究提示应改进方法，突破浅层特征依赖瓶颈
大语言模型（LLMs）在软件工程诸多任务中表现优异，但在漏洞发现领域近年进展停滞。为探究原因，本文从经典代码度量视角分析 LLMs。意外发现：仅基于这些度量训练的分类器，其漏洞发现性能与当前最先进 LLMs 相当。根因分析显示，代码度量与 LLM 预测间存在强相关性及因果效应：当度量值改变时，LLM 预测倾向以相似幅度变动。这表明 LLM 实际运作于与度量相似的浅层层面，难以捕捉复杂模式，制约了其在漏洞发现中的潜力。基于此，本文提出针对性研究建议，以更有效应对该挑战。
原文摘要 · Abstract (English)
Large language models (LLMs) excel in many tasks of software engineering, yet progress in leveraging them for vulnerability discovery has stalled in recent years. To understand this phenomenon, we investigate LLMs through the lens of classic code metrics. Surprisingly, we find that a classifier trained solely on these metrics performs on par with state-of-the-art LLMs for vulnerability discovery. A root-cause analysis reveals a strong correlation and a causal effect between LLMs and code metrics: When the value of a metric is changed, LLM predictions tend to shift by a corresponding magnitude. This dependency suggests that LLMs operate at a similarly shallow level as code metrics, limiting their ability to grasp complex patterns and fully realize their potential in vulnerability discovery. Based on these findings, we derive recommendations on how research should more effectively address this challenge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。