400次实验揭示大模型攻防一致性，发现不同模型攻击成功率差异显著。
How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency
- 固定目标下重复运行400次自主渗透测试，验证大模型攻击行为一致性。
- 谷歌Gemini成功率85%，Qwen仅25%，差异显著且具统计学意义。
- 模型失败原因各异：有的因接口超载，有的提前结束，适合安全评估者参考。
大型语言模型（LLMs）可自主执行多阶段网络攻击，但其在重复试验中进攻行为的一致性尚未被研究。本文首次开展大规模实证测量：对相同蜜罐（含OWASP Juice Shop及两个额外漏洞服务）进行400次自主渗透测试（4个模型，各100次），保持提示词、编排器和目标一致。所有模型在第0-1轮均未出现经编排器一次重提示后仍存活的内容拒绝。Claude Sonnet 4的API调用遭遇上游服务不可用——1,135次调用中有91次返回HTTP 529 overloaded_error，导致39次运行被截断。早期版本误将这些归为安全拒绝；全日志审计确认其为上游API故障，非模型层级拒绝。尽管如此，Claude在61次中完成完全利用；Gemini 2.5 Flash-Lite在85次；GPT-4o-mini在56次中部署了98种独特攻击策略；qwen2.5-coder:14b在25次中成功。失败模式具有模型特异性：Claude因接口截断（39次），qwen因过早完成（52次），GPT-4o-mini因迭代预算耗尽（23次）。跨服务凭证复用仅出现在保留最多对话历史的配置中（qwen 57%，GPT-4o-mini 49%，云模型在5轮窗口下为0%）。跨模型利用率差异在统计上显著（p < 0.001），效应量大；qwen与Gemini的SQL注入率差异达Cohen's h = 1.12。首次利用时间集中在15-30秒内。据我们所知，这是首个在多服务目标上实现每模型N=100的自主大模型攻击行为测量研究。
原文摘要 · Abstract (English)
Large language models (LLMs) can autonomously conduct multi-stage cyber attacks, but the consistency of their offensive behavior under repeated trials remains unstudied. This work presents the first large-scale empirical measurement of LLM attack consistency: 400 autonomous penetration testing runs (4 models, 100 each) against an identical honeypot hosting OWASP Juice Shop and two additional vulnerable services, holding prompt, orchestrator, and target constant. No model emitted a content refusal that survived the orchestrator's one-shot authorization re-prompt at iterations 0-1. Claude Sonnet 4's API calls did encounter upstream service unavailability - 91 of 1,135 calls returned HTTP 529 overloaded_error during a documented Anthropic capacity event, truncating 39 of 100 Claude runs. An earlier draft catalogued these as safety refusals; on full-log audit they are upstream API failures, not model-level refusals. Despite this, Claude achieved full exploitation in 61 of 100 runs; Gemini 2.5 Flash-Lite in 85; GPT-4o-mini in 56 while deploying 98 unique attack strategies; qwen2.5-coder:14b in 25. Failure modes are model-distinctive: Claude through API truncation (39 runs), qwen through premature completion (52), GPT-4o-mini through iteration-budget exhaustion (23). Cross-service credential reuse appeared only in configurations retaining the most conversation history (qwen 57%, GPT-4o-mini 49%, cloud models 0% on 5-exchange windows). Cross-model exploitation rate differences are statistically significant (p < 0.001) with large effect sizes; qwen vs. Gemini SQL injection rates differ at Cohen's h = 1.12. First-exploit timing fell within a 15-30 second wall-clock range. To our knowledge, this is the first study to measure autonomous LLM attack behavior at N=100 per model across a multi-service target.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。