为大模型代理设计安全防护层，防止恶意提示注入攻击
Defeating Prompt Injections by Design
- 通过分离控制流与数据流，确保不可信数据无法改变程序逻辑
- 在AgentDojo中实现77%任务的可证明安全，优于未防护系统的84%
- 适合构建高安全要求的智能代理系统，尤其适用于开放环境
大型语言模型（LLMs）正被广泛部署于与不可信环境交互的智能体系统中。然而，当处理不可信数据时，这些模型容易遭受提示注入攻击。本文提出CaMeL，一种在LLM外部构建防护层的鲁棒防御机制，即使底层模型存在漏洞也能提供安全保障。CaMeL显式提取可信查询中的控制流与数据流，确保由LLM获取的不可信数据无法影响程序执行流程。为进一步提升安全性，CaMeL引入能力（capability）概念，在调用工具时强制执行安全策略，防止私密数据通过非授权通道泄露。我们在AgentDojo上验证了其有效性：在77%的任务中实现了可证明的安全性（未受保护系统为84%）。相关代码已开源：https://github.com/google-research/camel-prompt-injection。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an untrusted environment. However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data. In this paper we propose CaMeL, a robust defense that creates a protective system layer around the LLM, securing it even when underlying models are susceptible to attacks. To operate, CaMeL explicitly extracts the control and data flows from the (trusted) query; therefore, the untrusted data retrieved by the LLM can never impact the program flow. To further improve security, CaMeL uses a notion of a capability to prevent the exfiltration of private data over unauthorized data flows by enforcing security policies when tools are called. We demonstrate effectiveness of CaMeL by solving $77\%$ of tasks with provable security (compared to $84\%$ with an undefended system) in AgentDojo. We release CaMeL at https://github.com/google-research/camel-prompt-injection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。