arXiv:2502.05174cs.CRcs.AI2025-02ICML被引 72

MELON通过重执行检测间接提示注入攻击,兼顾安全与功能可用性。

MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents

  • 通过掩码用户提示后重执行轨迹,判断动作是否受恶意任务影响。
  • 在AgentDojo基准上,攻击防御率超现有方法,且正常功能损失更小。
  • 适合需要高安全性但又不能牺牲实用性的AI代理系统开发者。

近期研究发现,大语言模型代理易受间接提示注入(IPI)攻击,即工具检索信息中嵌入的恶意任务可引导代理执行未经授权的操作。现有防御存在明显局限:或需大量训练资源,或对复杂攻击无效,或损害正常功能。本文提出MELON(Masked re-Execution and TooL comparisON),一种新型IPI防御机制。其核心思想是:成功攻击下,代理下一步动作对用户任务的依赖降低,转而受恶意任务主导。MELON通过掩码函数修改用户提示后重执行代理轨迹,若原始与掩码执行的动作相似,则判定为攻击。设计中包含三项优化以减少误报与漏报。在IPI基准AgentDojo上的评估表明,MELON优于当前最优防御,在攻击阻断与功能保留方面均表现更优。进一步将MELON与最优提示增强防御(MELON-Aug)结合,性能再提升。详细消融实验验证了关键设计的有效性。代码已开源:https://github.com/kaijiezhu11/MELON。

原文摘要 · Abstract (English)

Recent research has explored that LLM agents are vulnerable to indirect prompt injection (IPI) attacks, where malicious tasks embedded in tool-retrieved information can redirect the agent to take unauthorized actions. Existing defenses against IPI have significant limitations: either require essential model training resources, lack effectiveness against sophisticated attacks, or harm the normal utilities. We present MELON (Masked re-Execution and TooL comparisON), a novel IPI defense. Our approach builds on the observation that under a successful attack, the agent's next action becomes less dependent on user tasks and more on malicious tasks. Following this, we design MELON to detect attacks by re-executing the agent's trajectory with a masked user prompt modified through a masking function. We identify an attack if the actions generated in the original and masked executions are similar. We also include three key designs to reduce the potential false positives and false negatives. Extensive evaluation on the IPI benchmark AgentDojo demonstrates that MELON outperforms SOTA defenses in both attack prevention and utility preservation. Moreover, we show that combining MELON with a SOTA prompt augmentation defense (denoted as MELON-Aug) further improves its performance. We also conduct a detailed ablation study to validate our key designs. Code is available at https://github.com/kaijiezhu11/MELON.

AI安全提示注入防御机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。