arXiv:2506.13205cs.CRcs.AI2025-06被引 1

用视觉触发控制手机智能体,不改文字就能植入后门

Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents

  • 仅修改图像输入,不改动文本指令实现隐蔽后门
  • 攻击成功率最高达94.67%,正常任务准确率仍保持95.85%
  • 适用于研究移动智能体安全的开发者与防御研究人员

基于视觉语言模型(VLM)的移动智能体在用户界面自动化和基于摄像头的辅助等任务中日益普及。这些智能体通常使用少量用户收集的数据进行微调,易受训练阶段隐蔽威胁。本文提出VIBMA,首个针对基于VLM的移动智能体的干净文本后门攻击。该攻击仅通过修改视觉输入注入恶意行为,同时保持文本提示和指令不变,实现完全无文本异常的隐蔽性。一旦智能体在污染数据上微调,推理时加入预设视觉模式(触发器)即可激活攻击者指定行为(后门)。攻击通过对齐污染样本的训练梯度与目标实例梯度,将后门特征嵌入污染数据。为增强攻击鲁棒性和隐蔽性,设计三种更贴近真实场景的触发变体:静态贴图、动态运动模式和低透明度融合内容。在六个Android应用和三个移动端兼容VLM上的实验表明,该攻击在保持高清洁任务准确率的同时,实现高达94.67%的攻击成功率(ASR)和95.85%的正常任务保真率(FSR)。消融研究揭示关键设计因素对攻击可靠性和隐蔽性的影响。这是首次揭示移动智能体在适应过程中的安全漏洞,凸显其对后门注入的高度敏感性,亟需建立稳健的防御机制。

原文摘要 · Abstract (English)

Mobile agents powered by vision-language models (VLMs) are increasingly adopted for tasks such as UI automation and camera-based assistance. These agents are typically fine-tuned using small-scale, user-collected data, making them susceptible to stealthy training-time threats. This work introduces VIBMA, the first clean-text backdoor attack targeting VLM-based mobile agents. The attack injects malicious behaviors into the model by modifying only the visual input while preserving textual prompts and instructions, achieving stealth through the complete absence of textual anomalies. Once the agent is fine-tuned on this poisoned data, adding a predefined visual pattern (trigger) at inference time activates the attacker-specified behavior (backdoor). Our attack aligns the training gradients of poisoned samples with those of an attacker-specified target instance, effectively embedding backdoor-specific features into the poisoned data. To ensure the robustness and stealthiness of the attack, we design three trigger variants that better resemble real-world scenarios: static patches, dynamic motion patterns, and low-opacity blended content. Extensive experiments on six Android applications and three mobile-compatible VLMs demonstrate that our attack achieves high success rates (ASR up to 94.67%) while preserving clean-task behavior (FSR up to 95.85%). We further conduct ablation studies to understand how key design factors impact attack reliability and stealth. These findings is the first to reveal the security vulnerabilities of mobile agents and their susceptibility to backdoor injection, underscoring the need for robust defenses in mobile agent adaptation pipelines.

后门攻击移动智能体VLM安全视觉触发

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。