用少量示例引导智能体先防御后行动,大幅降低欺骗攻击成功率。
In-Context Defense in Computer Agents: An Empirical Study
- 在上下文中加入防御性示例,引导智能体先推理再执行。
- 对弹窗攻击有效率提升91.2%,对环境注入攻击平均提升74.6%。
- 仅需不到三个示例即可触发防御行为,适合部署于实际系统。
由视觉语言模型(VLMs)驱动的计算机智能体显著提升了人机交互能力,使用户可通过自然语言指令完成复杂任务。然而,这些智能体易受上下文欺骗攻击,即攻击者将误导内容嵌入操作环境(如含虚假指令的弹窗)。现有防御策略(如要求忽略欺骗元素)效果有限。作为首个针对计算机智能体保护的系统性研究,本文提出「上下文防御」机制,利用上下文学习与链式思维(CoT)推理,通过在智能体上下文中添加少量精心设计的示例(包含恶意环境与对应防御响应),引导其在行动规划前进行显式防御推理,从而降低被欺骗风险。实验表明,该方法在弹窗攻击中将攻击成功率降低91.2%,在环境注入攻击中平均降低74.6%,并实现对干扰广告100%的有效防御。研究发现:(1) 防御推理必须先于行动规划;(2) 少量示例(少于三个)即可诱导智能体产生防御行为。
原文摘要 · Abstract (English)
Computer agents powered by vision-language models (VLMs) have significantly advanced human-computer interaction, enabling users to perform complex tasks through natural language instructions. However, these agents are vulnerable to context deception attacks, an emerging threat where adversaries embed misleading content into the agent's operational environment, such as a pop-up window containing deceptive instructions. Existing defenses, such as instructing agents to ignore deceptive elements, have proven largely ineffective. As the first systematic study on protecting computer agents, we introduce textbf{in-context defense}, leveraging in-context learning and chain-of-thought (CoT) reasoning to counter such attacks. Our approach involves augmenting the agent's context with a small set of carefully curated exemplars containing both malicious environments and corresponding defensive responses. These exemplars guide the agent to first perform explicit defensive reasoning before action planning, reducing susceptibility to deceptive attacks. Experiments demonstrate the effectiveness of our method, reducing attack success rates by 91.2% on pop-up window attacks, 74.6% on average on environment injection attacks, while achieving 100% successful defenses against distracting advertisements. Our findings highlight that (1) defensive reasoning must precede action planning for optimal performance, and (2) a minimal number of exemplars (fewer than three) is sufficient to induce an agent's defensive behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。