测试九种防御方案后发现，唯有应用层过滤能完全防止大模型泄露密钥。
Evaluation of Prompt Injection Defenses in Large Language Models
- 用自适应攻击者模拟数百轮试探，持续进化攻击策略。
- 9种依赖模型自身防护的方案全部失效，仅输出过滤零泄露。
- 建议敏感任务由可信人员操作，防御需在应用代码中实现。
基于大语言模型的应用常将密钥嵌入系统提示词，但模型可能被诱导泄露。我们构建了一个自适应攻击者，在超过20,000次攻击中迭代演化策略，并测试了九种防御配置。所有依赖模型自我保护的方案最终均被突破。唯一有效的防御是输出过滤——通过独立应用代码中的硬编码规则检查模型响应，该方法在15,000次攻击中实现零泄露。结果表明，安全边界必须在应用代码中强制执行，而非依赖被攻击的模型。在Swept AI等工具验证前，处理敏感操作的AI系统应限制为内部可信人员使用。
原文摘要 · Abstract (English)
LLM-powered applications routinely embed secrets in system prompts, yet models can be tricked into revealing them. We built an adaptive attacker that evolves its strategies over hundreds of rounds and tested it against nine defense configurations across more than 20,000 attacks. Every defense that relied on the model to protect itself eventually broke. The only defense that held was output filtering, which checks the model's responses via hardcoded rules in separate application code before they reach the user, achieving zero leaks across 15,000 attacks. These results demonstrate that security boundaries must be enforced in application code, not by the model being attacked. Until such defenses are verified by tools like Swept AI, AI systems handling sensitive operations should be restricted to internal, trusted personnel.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。