用误导性回应阻断自动化攻击,显著降低越狱成功率。
Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systems

- 用可控误导回复替代固定拒绝,干扰攻击者判断
- 在基准测试中将攻击成功上限降低两个数量级
- 适合防御自动化越狱攻击的智能系统使用
智能体系统依赖语言模型解析指令、处理数据、调用工具并协同其他代理,使其易受提示注入和越狱攻击影响。随着攻击者采用模型引导的自动化手段扩大探测范围、优化提示并评估响应,威胁加剧。本文通过概率模型分析攻防态势,发现传统检测拦截机制因可预测的拒绝反馈,导致攻击成功率随查询预算增长趋近于1。提出“检测-误导”策略:对恶意交互返回非操作性、有目的的误导响应,降低攻击者判断的正向预测价值,实现攻击成功率的有界收敛。验证了概念原型「渐进式参与上下文误导」(CMPE),一种轻量级对话误导方法,在自动越狱场景中以安全但策略性误导的回复替代固定拒绝。在越狱基准测试中,CMPE将估计攻击成功率上限降低至原值的1%,并在端到端PAIR与GPTFuzz攻击中几乎消除实际成功案例。
原文摘要 · Abstract (English)
Agentic AI systems increasingly rely on language-model components to interpret instructions, process external data, invoke tools, and coordinate with other agents. These capabilities make prompt-injection and jailbreak attacks more consequential, especially as attackers adopt model-guided automation to scale probing, prompt refinement, and response evaluation. This work analyzes the resulting attack-defense setting through a probabilistic model of a target system, its defense mechanism, and the attacker's automated judge. Our analysis shows that conventional detect-and-block defenses can allow attacker success rate (ASR) to approach one as the query budget grows, since predictable refusals provide useful feedback to automated search. We then examine detect-and-misdirect, where detected malicious interactions receive controlled, non-operational responses designed to induce false-positive errors in the attacker's judge. This strategy reduces the positive predictive value of attacker-selected candidates and yields a bounded asymptotic ASR. We evaluate a proof-of-concept realization of this strategy through Contextual Misdirection via Progressive Engagement (CMPE), a lightweight conversational misdirection method designed to replace predictable refusal text with safe but strategically misleading responses in automated jailbreak settings. On jailbreak benchmarks, CMPE reduces estimated ASR upper bounds by up to two orders of magnitude and nearly eliminates verified attack success in end-to-end PAIR and GPTFuzz attack runs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。