arXiv:2512.00966cs.CRcs.LG2025-12被引 12

通过分析模型是否意图执行恶意指令,有效防御隐藏在输入中的攻击。

Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysis

  • 识别输入中模型真正打算执行的指令,而非仅看是否有恶意文本。
  • 在两个基准测试中,攻击成功率从100%降至8.5%,且基本不损失原有功能。
  • 适合开发智能代理或需高安全性的LLM应用者使用。

间接提示注入攻击(IPIAs)威胁大型语言模型(LLMs)驱动的智能体安全。本文提出基于指令遵循意图分析的通用防御框架IntentGuard。核心洞察是:攻击成败关键不在于恶意文本是否存在,而在于模型是否意图执行来自不可信数据的指令。IntentGuard利用指令遵循意图分析器(IIA)识别输入中被模型视为可执行指令的部分,并标记或中和其与不可信数据的重叠。为实现该框架,我们设计了一种采用三种“思维干预”策略的IIA:思考开始前填充、思考结束后精炼、对抗性上下文示范。在AgentDojo和Mind2Web两个智能体基准上,使用Qwen-3-32B和gpt-oss-20B两种推理型LLM进行评估。结果表明,IntentGuard在几乎所有设置下均无性能下降,且对自适应提示注入攻击表现出强鲁棒性(如在Mind2Web场景中,将攻击成功率从100%降低至8.5%)。

原文摘要 · Abstract (English)

Indirect prompt injection attacks (IPIAs), where large language models (LLMs) follow malicious instructions hidden in input data, pose a critical threat to LLM-powered agents. In this paper, we present IntentGuard, a general defense framework based on instruction-following intent analysis. The key insight of IntentGuard is that the decisive factor in IPIAs is not the presence of malicious text, but whether the LLM intends to follow instructions from untrusted data. Building on this insight, IntentGuard leverages an instruction-following intent analyzer (IIA) to identify which parts of the input prompt the model recognizes as actionable instructions, and then flag or neutralize any overlaps with untrusted data segments. To instantiate the framework, we develop an IIA that uses three "thinking intervention" strategies to elicit a structured list of intended instructions from reasoning-enabled LLMs. These techniques include start-of-thinking prefilling, end-of-thinking refinement, and adversarial in-context demonstration. We evaluate IntentGuard on two agentic benchmarks (AgentDojo and Mind2Web) using two reasoning-enabled LLMs (Qwen-3-32B and gpt-oss-20B). Results demonstrate that IntentGuard achieves (1) no utility degradation in all but one setting and (2) strong robustness against adaptive prompt injection attacks (e.g., reducing attack success rates from 100% to 8.5% in a Mind2Web scenario).

安全防护提示注入智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。