arXiv:2511.20223cs.CV2025-11中稿 · CVPR被引 7

通过操控视觉语言模型的值特征,实现对图像局部语义的精准攻击。

V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs

  • 瞄准变压器中值特征,避开纠缠的补丁特征以提升控制精度。
  • 在多个主流模型上平均攻击成功率提升36%。
  • 适合研究模型安全与对抗鲁棒性的研究人员使用。

对抗攻击已从干扰传统任务模型预测,演变为操控大型视觉语言模型(LVLMs)的图像语义。然而,现有方法难以实现可控性,无法精确操纵图像中特定概念的语义。我们发现,攻击通常作用于视觉编码器自注意力聚合全局上下文的补丁-令牌表示,导致局部特征被语义纠缠所掩盖,难以用于精准操作。系统研究表明,变压器注意力模块中的值特征(V)保留了更高熵、解耦的局部语义信息,抑制了全局上下文通道,是更优的操作对象。基于此,我们提出V-Attack,其核心为:(1) 自值增强模块以强化值特征的内在语义丰富度;(2) 文本引导值操纵模块,利用文本提示定位源概念并优化其向目标概念迁移。该方法绕过纠缠的补丁特征,实现高效语义控制。在LLaVA、InternVL、DeepseekVL和GPT-4o等多样化的LVLM上进行的大量实验表明,相比最先进方法,V-Attack平均攻击成功率提升36%,揭示了现代视觉语言理解模型的关键脆弱性。代码与数据已公开于https://github.com/Summu77/V-Attack。

原文摘要 · Abstract (English)

Adversarial attacks have evolved from simply disrupting predictions on conventional task-specific models to the more complex goal of manipulating image semantics on Large Vision-Language Models (LVLMs). However, existing methods struggle with controllability and fail to precisely manipulate the semantics of specific concepts in the image. We attribute this limitation to semantic entanglement in the patch-token representations on which adversarial attacks typically operate: global context aggregated by self-attention in the vision encoder dominates individual patch features, making them unreliable handles for precise local semantic manipulation. Our systematic investigation reveals a key insight: value features (V) computed within the transformer attention block serve as much more precise handles for manipulation. We show that V suppresses global-context channels, allowing it to retain high-entropy, disentangled local semantic information. Building on this discovery, we propose V-Attack, a novel method designed for precise local semantic attacks. V-Attack targets the value features and introduces two core components: (1) a Self-Value Enhancement module to refine V's intrinsic semantic richness, and (2) a Text-Guided Value Manipulation module that leverages text prompts to locate source concept and optimize it toward a target concept. By bypassing the entangled patch features, V-Attack achieves highly effective semantic control. Extensive experiments across diverse LVLMs, including LLaVA, InternVL, DeepseekVL and GPT-4o, show that V-Attack improves the attack success rate by an average of 36% over state-of-the-art methods, exposing critical vulnerabilities in modern visual-language understanding. Our code and data are available https://github.com/Summu77/V-Attack.

对抗攻击视觉语言模型语义控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。