通过原型与注意力机制，高效攻击多模型视觉编码器。
PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention
- 用原型引导攻击方向，提升跨任务泛化能力。
- 两阶段注意力增强使扰动聚焦关键视觉特征，平均攻击成功率提升75.1%。
- 适用于研究模型鲁棒性或安全评估的开发者。
大型视觉语言模型（LVLMs）是现代多模态应用的基础，但其易受对抗攻击的特性仍是关键挑战。以往白盒攻击泛化能力差，黑盒方法依赖昂贵的迁移，效率受限。视觉编码器作为标准化组件，常被多个LVLM共享，提供了稳定的灰盒攻击切入点。本文提出PA-Attack（原型锚定注意力攻击），首先采用原型锚定引导，生成稳定且通用的攻击方向，解决传统攻击属性受限与任务泛化不足问题；在此基础上，设计两阶段注意力增强机制：(i) 利用令牌级注意力分数，将扰动集中于关键视觉令牌；(ii) 自适应重校准注意力权重，追踪对抗过程中的注意力演化。在多种下游任务和LVLM架构上的大量实验表明，PA-Attack实现了平均75.1%的得分降低率（SRR），展现出强攻击效果、高效率与良好任务泛化能力。代码已开源：https://github.com/hefeimei06/PA-Attack。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) are foundational to modern multimodal applications, yet their susceptibility to adversarial attacks remains a critical concern. Prior white-box attacks rarely generalize across tasks, and black-box methods depend on expensive transfer, which limits efficiency. The vision encoder, standardized and often shared across LVLMs, provides a stable gray-box pivot with strong cross-model transfer. Building on this premise, we introduce PA-Attack (Prototype-Anchored Attentive Attack). PA-Attack begins with a prototype-anchored guidance that provides a stable attack direction towards a general and dissimilar prototype, tackling the attribute-restricted issue and limited task generalization of vanilla attacks. Building on this, we propose a two-stage attention enhancement mechanism: (i) leverage token-level attention scores to concentrate perturbations on critical visual tokens, and (ii) adaptively recalibrate attention weights to track the evolving attention during the adversarial process. Extensive experiments across diverse downstream tasks and LVLM architectures show that PA-Attack achieves an average 75.1% score reduction rate (SRR), demonstrating strong attack effectiveness, efficiency, and task generalization in LVLMs. Code is available at https://github.com/hefeimei06/PA-Attack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。