SIR通过自迭代优化,让电脑操作代理更易受隐蔽攻击
SIR: Self-improving Red-teaming for Compute Use Agents

- 用可复用的自然语言原则+反馈循环生成隐蔽攻击
- 对Claude Opus成功率从4%提至24%,Gemini达28%
- 发现的攻击策略可跨模型迁移,适合安全评估者参考
计算机使用代理(CUAs)是能感知屏幕并操作真实系统的视觉语言模型,日益用于自动化数字任务。由于可能接触不可信内容,它们易受间接提示注入(IPI)攻击,即攻击者在代理将读取的内容中植入指令,使其偏离用户意图。现有安全基准仅测试人工编写的固定注入,可能低估适应性攻击的风险。本文提出SIR,一种黑盒IPI攻击方法:(i) 从少量自然语言表述的可复用原则库中组合隐蔽注入;(ii) 在迭代反馈回路中诊断失败轨迹,并将绕过策略提炼为新命名策略,跨任务重用。不同于以往针对网页代理的红队测试,SIR在操作系统层面攻击CUAs,使用确定性检测器(检查文件系统、服务与权限状态)而非大模型判断。实验评估三个前沿CUAs:基于反馈的原则组合使攻击成功率显著提升,例如在Claude Opus 4.8上从4%升至24%,在Gemini 3.5 Flash上从0%升至28%,且良性任务仍正常完成。针对某一模型发现的策略可无需额外反馈地迁移至不同架构。
原文摘要 · Abstract (English)
Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal, and they are increasingly deployed to automate everyday digital tasks. Because they can be exposed to untrusted content while operating, they are vulnerable to indirect prompt injection (IPI), in which an adversary plants instructions in content the agent will read and redirects it toward actions that violate the user's intent. Existing CUA safety benchmarks evaluate fixed injections written by hand, which may underestimate the risk posed by an adaptive adversary. We present SIR, a black box IPI attack that (i) composes stealthy injections from a small library of reusable principles stated in plain language and (ii) wraps composition in an iterative feedback loop that diagnoses the victim's failed trajectories and distills the bypasses into new, named strategies that are reapplied across tasks. Unlike prior red teaming of web agents, we target CUAs at the operating system level and score attacks with a fully deterministic oracle, using checks on filesystem, service, and permission state rather than an LLM judge. On experiment, we evaluate three frontier CUAs. Composing principles with feedback raises the attack success rate over a baseline written by hand, for example from 4% to 24% on Claude Opus 4.8 and from 0% to 28% on Gemini 3.5 Flash, while the benign task still completes. Principles discovered against one model further transfer to a different architecture with no additional feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。