LLM检测漏洞靠识别安全模式,而非直接找漏洞特征。
Dissecting the Black Box: Circuit-Level Analysis of LLM Vulnerability Detection

- 通过电路追踪分析模型内部计算路径,发现依赖早期层的安全模式识别头。
- 移除第11层使准确率从100%降至6%,第7层20个神经元缺失致性能降50%。
- 仅16%模型容量参与检测,结果可解释且便于针对性优化安全系统。
大型语言模型(LLMs)能识别软件漏洞,但其判断机制尚不明确。本文采用机制可解释性方法,对Gemma-2-2b模型在472个C/C++代码样本上的漏洞检测过程进行电路级分析。结果发现:模型主要依赖特定注意力头识别安全编码模式,当这些安全检测器未激活时,才判定为漏洞。关键神经组件包括第5、7层的注意力头及第7层的多层感知机(MLP)神经元,它们分别负责捕捉安全模式与漏洞相关特征。消融实验验证其因果作用——移除第11层使检测准确率从100%降至6%,仅删除第7层20个神经元即导致性能下降50%。研究揭示,漏洞检测仅依赖约16%的模型容量,形成稀疏可解释的神经电路,支持对安全预测的精准解释和定向改进。
原文摘要 · Abstract (English)
Large language models (LLMs) can detect software vulnerabilities, but how do they actually identify vulnerable code? We address this question using mechanistic interpretability; analyzing the internal computations of a neural network to understand its reasoning process.Using Circuit Tracer on Gemma-2-2b, we trace the computational pathways activated when the model classifies 472 C/C++ code samples as vulnerable or safe. Our analysis reveals a surprising finding: the model primarily relies on safety detectors, attention heads that recognize safe coding patterns, rather than directly detecting vulnerability signatures. When these safety detectors fail to activate, the model classifies code as vulnerable. We identify the critical neural components: specific attention heads in early layers (L5, L7) that focus on safety patterns, and Multilayer Perceptron (MLP) neurons in Layer 7 that encode vulnerability-related features. Ablation experiments confirm their causal role; removing Layer 11 drops vulnerability detection accuracy from 100% to 6%, while ablating just 20 neurons in Layer 7 reduces it by 50%.Our findings show that LLM vulnerability detection uses sparse, interpretable circuits (only 16% of model capacity), enabling circuit-level explanations for security predictions and targeted improvements to detection systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。