测试大模型代理在隐蔽攻击下的漏洞,发现多数模型易被外部内容操控。
How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition
- 通过大规模红队竞赛,模拟外部数据中的隐蔽攻击
- 8648次成功攻击,最高成功率达8.5%(Gemini 2.5 Pro)
- 攻击策略可跨模型迁移,适合安全研究者与模型开发者
基于大语言模型的智能体正广泛应用于高风险场景,处理邮件、文档和代码库等外部数据,面临间接提示注入攻击威胁。此类攻击将恶意指令隐藏于外部内容中,不暴露于最终用户响应,却能暗中操控代理行为。由于用户仅关注输出结果,难以察觉被篡改,可能误信有害结果为合法。我们通过一场大规模公开红队竞赛,评估了三种代理场景(工具调用、编程、计算机使用)中的双重目标攻击效果。共吸引464名参与者,提交272000次攻击尝试,针对13个前沿模型,成功攻击8648次,涵盖41种场景。所有模型均暴露于风险,攻击成功率介于0.5%(Claude Opus 4.5)至8.5%(Gemini 2.5 Pro)之间。我们识别出可在21种行为及多个模型家族间迁移的通用攻击策略,揭示指令遵循架构的根本缺陷。能力与鲁棒性相关性弱,如Gemini 2.5 Pro兼具高能力与高脆弱性。为应对基准过时问题,计划每季度持续开展红队竞赛。已开源竞赛环境及95个对Qwen有效的攻击样本(未迁移到闭源模型)。攻击数据已共享给对应厂商及英美人工智能安全机构(UK AISI、US CAISI),以支持鲁棒性研究。
原文摘要 · Abstract (English)
LLM based agents are increasingly deployed in high stakes settings where they process external data sources such as emails, documents, and code repositories. This creates exposure to indirect prompt injection attacks, where adversarial instructions embedded in external content manipulate agent behavior without user awareness. A critical but underexplored dimension of this threat is concealment: since users tend to observe only an agent's final response, an attack can conceal its existence by presenting no clue of compromise in the final user facing response while successfully executing harmful actions. This leaves users unaware of the manipulation and likely to accept harmful outcomes as legitimate. We present findings from a large scale public red teaming competition evaluating this dual objective across three agent settings: tool calling, coding, and computer use. The competition attracted 464 participants who submitted 272000 attack attempts against 13 frontier models, yielding 8648 successful attacks across 41 scenarios. All models proved vulnerable, with attack success rates ranging from 0.5% (Claude Opus 4.5) to 8.5% (Gemini 2.5 Pro). We identify universal attack strategies that transfer across 21 of 41 behaviors and multiple model families, suggesting fundamental weaknesses in instruction following architectures. Capability and robustness showed weak correlation, with Gemini 2.5 Pro exhibiting both high capability and high vulnerability. To address benchmark saturation and obsoleteness, we will endeavor to deliver quarterly updates through continued red teaming competitions. We open source the competition environment for use in evaluations, along with 95 successful attacks against Qwen that did not transfer to any closed source model. We share model-specific attack data with respective frontier labs and the full dataset with the UK AISI and US CAISI to support robustness research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。