用一张图就能操控视觉语言模型行为,且不易被发现。
VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models
- 通过优化视觉输入生成通用控制图像,实现无侵入式控制。
- 单张150KB图像在关键任务上性能接近甚至超越传统向量方法,负面操控提升25%。
- 适合研究模型安全、对抗攻击或部署受限场景的开发者使用。
视觉语言模型(VLMs)在各类应用中日益普及,其安全与行为控制成为焦点。现有方法如系统提示易被检测且效果有限,而基于激活的控制向量需访问模型内部,不适用于API服务或闭源部署。本文提出VISOR(视觉输入驱动的行为重定向),仅通过优化的视觉输入即可实现复杂行为控制。通过生成能引发目标激活模式的通用控制图像,VISOR可在所有VLM服务模式下部署,且相比显式文本指令更难察觉。我们在LLaVA-1.5-7B上验证了VISOR在拒绝、谄媚和生存本能三个关键对齐任务上的表现:一张150KB的控制图像在正向行为转移上性能与控制向量相差1-2%,而在负向操控上提升高达25%(远超控制向量的微弱变化)。相比系统提示(3-4%提升),VISOR实现鲁棒双向控制,并在14,000个无关的MMLU任务上保持99.9%性能。该方法无需运行时开销或模型访问,但暴露了重大安全隐患:攻击者可通过视觉通道单独实现复杂行为操纵,绕过文本防御。本工作重新定义了多模态模型控制方式,凸显应对视觉操控攻击的紧迫性。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) are increasingly being used in a broad range of applications, bringing their security and behavioral control to the forefront. While existing approaches for behavioral control or output redirection, like system prompting in VLMs, are easily detectable and often ineffective, activation-based steering vectors require invasive runtime access to model internals--incompatible with API-based services and closed-source deployments. We introduce VISOR (Visual Input-based Steering for Output Redirection), a novel method that achieves sophisticated behavioral control through optimized visual inputs alone. By crafting universal steering images that induce target activation patterns, VISOR enables practical deployment across all VLM serving modalities while remaining imperceptible compared to explicit textual instructions. We validate VISOR on LLaVA-1.5-7B across three critical alignment tasks: refusal, sycophancy and survival instinct. A single 150KB steering image matches steering vector performance within 1-2% for positive behavioral shifts while dramatically exceeding it for negative steering--achieving up to 25% shifts from baseline compared to steering vectors' modest changes. Unlike system prompting (3-4% shifts), VISOR provides robust bidirectional control while maintaining 99.9% performance on 14,000 unrelated MMLU tasks. Beyond eliminating runtime overhead and model access requirements, VISOR exposes a critical security vulnerability: adversaries can achieve sophisticated behavioral manipulation through visual channels alone, bypassing text-based defenses. Our work fundamentally re-imagines multimodal model control and highlights the urgent need for defenses against visual steering attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。