测试LLM找漏洞的重复性，发现结果不稳定，需结合传统检测工具。
Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?
- 用300次重复扫描测试LLM漏洞检测一致性，对比不同模型表现。
- 80个非匹配发现仅在1次中出现，而参考匹配发现90%以上稳定重现。
- 建议将LLM与确定性SAST工具结合，提升漏洞检测可靠性。
我们对同一份JavaScript代码、相同提示和基准测试框架进行了300次重复漏洞扫描，评估代理型大语言模型（LLM）在安全审查中的可重复性。主要发现是：LLM的安全检测结果不一致——与参考匹配的发现较稳定，但额外报告在各次运行间差异显著。在250次模型运行中，161个唯一未匹配发现中有80个仅在五次重复中的一次出现，仅有22个出现在全部五次。相比之下，当Claude匹配了Snyk Code参考发现时，行为更稳定：158个唯一匹配发现中有134个出现在所有五次重复中。该基准还揭示出互补性：模型持续发现熟悉的高信号攻击模式，并在一处暴露了可能的Snyk Code产品缺口。Snyk Code静态应用安全测试（SAST）具有确定性，更擅长系统性枚举重复的数据流汇聚点。结果表明，应将代理型LLM审查与确定性SAST结合，而非相互替代。
原文摘要 · Abstract (English)
We ran 300 repeated vulnerability-finding scans to measure how repeatable agentic large language model (LLM) security review is on the same JavaScript code, prompt, and benchmark harness. The headline result is that LLM security findings were unevenly repeatable: reference-matched findings were stable, but extra model reports varied heavily from run to run. Across 250 model runs, 80 of 161 unique unmatched findings appeared in only one of five identical repetitions, while only 22 appeared in all five. By contrast, when Claude matched a Snyk Code reference finding, the behavior was much more stable: 134 of 158 unique reference-matched findings appeared in all five repetitions. The benchmark also shows complementarity. Models consistently found familiar, high-signal exploit shapes, and in one case surfaced a likely Snyk Code product gap. Snyk Code static application security testing (SAST) was deterministic and better at systematically enumerating repeated data-flow sinks. The results support combining agentic LLM review with deterministic SAST rather than treating either technique as a replacement for the other.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。