通过逐标记净化指令,精准防御AI代理的间接提示注入攻击。
CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitization
- 在标记级别切除针对AI系统的指令,而非整体分类样本
- 在多个基准上将攻击成功率从34%降至3%,降低7-10倍
- 无需调参、不阻断流程,适合真实场景部署
随着大语言模型代理接入大量工具和敏感数据,间接提示注入攻击的威胁面显著扩大。由于攻击具有上下文依赖性,现有防御手段常因无法可靠区分恶意与良性指令,导致误报率过高,难以落地。为此,我们提出一种受计算机安全原理启发的新方法:数据不应包含可执行指令。不同于样本级分类,该方法采用标记级净化,精准移除工具输出中指向AI系统的指令,将其作为副产品捕获。此方法非阻塞、无需校准,且对工具输出上下文无感。此外,仅需通用指令微调数据即可训练标记级预测器,无需依赖虚构或挑战赛中的合成注入样本。实验表明,该方法在AgentDojo、BIPIA、InjecAgent、ASB和SEP等多类攻击与基准上表现良好,攻击成功率(ASR)从34%降至3%,降幅达7-10倍,且在正常与恶意场景下均未影响代理功能。
原文摘要 · Abstract (English)
The increasing adoption of LLM agents with access to numerous tools and sensitive data significantly widens the attack surface for indirect prompt injections. Due to the context-dependent nature of attacks, however, current defenses are often ill-calibrated as they cannot reliably differentiate malicious and benign instructions, leading to high false positive rates that prevent their real-world adoption. To address this, we present a novel approach inspired by the fundamental principle of computer security: data should not contain executable instructions. Instead of sample-level classification, we propose a token-level sanitization process, which surgically removes any instructions directed at AI systems from tool outputs, capturing malicious instructions as a byproduct. In contrast to existing safety classifiers, this approach is non-blocking, does not require calibration, and is agnostic to the context of tool outputs. Further, we can train such token-level predictors with readily available instruction-tuning data only, and don't have to rely on unrealistic prompt injection examples from challenges or of other synthetic origin. In our experiments, we find that this approach generalizes well across a wide range of attacks and benchmarks like AgentDojo, BIPIA, InjecAgent, ASB and SEP, achieving a 7-10x reduction of attack success rate (ASR) (34% to 3% on AgentDojo), without impairing agent utility in both benign and malicious settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。