通过关注热点区域逐步攻击,让小改动骗过视觉语言模型。
Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Models
- 用跨模态注意力识别关键区域,按重要性顺序更新
- 在10个模型上攻击成功率领先,且扰动更难察觉
- 无需目标模型参数,适合黑盒攻击场景
针对大型视觉语言模型(LVLMs)的定向对抗攻击,旨在验证微小图像扰动是否能引导模型输出攻击者指定内容。在标准L-infinity约束下,定向攻击本质上是区域扰动预算分配问题:攻击成功率不仅取决于目标,还与哪些区域被更新以及更新顺序有关。现有局部攻击虽优于全局扰动,但依赖随机空间采样,常更新影响较弱的区域。本文通过注意力分析发现,跨模态注意力能识别对抗敏感区域,且对高注意力热点区域扰动会引发后续显著区域的可预测重分布。基于此,提出分阶段注意力引导攻击(SAGA),一种无需访问目标模型参数、梯度或注意力图的黑盒区域序列化框架,利用开源LVLM的固定注意力图指导扰动更新。在10个闭源与开源自研LVLM上,SAGA实现当前最优攻击成功率与最佳整体不可察觉性。
原文摘要 · Abstract (English)
Targeted adversarial attacks on Large Vision-Language Models (LVLMs) test whether small image perturbations can steer model responses toward attacker-specified content. Under the standard L-infinity constraint, targeted attacks become a regional perturbation budget allocation problem: attack success depends not only on the perturbation objective, but also on which regions receive updates and in what order. Existing localized attacks improve over global perturbations but rely on stochastic spatial sampling, often updating weakly influential regions. We address this limitation through an attention-based analysis showing that cross-modal attention identifies adversarially sensitive regions and that perturbing high-attention hotspots induces predictable redistribution toward subsequent salient regions. These findings motivate attention-guided region sequencing, which begins from dominant hotspots and progressively moves the update support toward next-salient regions. Based on these principles, we propose Stage-wise Attention-Guided Attack (SAGA), a black-box region-sequencing framework that uses a fixed attention map from an open-source LVLM to guide perturbation updates without accessing target-model parameters, gradients, or attention maps. Across ten closed-source and open-source LVLMs, SAGA achieves state-of-the-art attack success rates and the best overall imperceptibility. The source code is available at https://github.com/jaehyun-kwak/SAGA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。