无需模型内部信息，用优化方法生成能骗过视觉语言大模型的隐蔽攻击输入。
Crafting Adversarial Inputs for Large Vision-Language Models Using Black-Box Optimization
- 基于零阶优化的黑箱攻击，仅通过输入输出互动估算梯度。
- 在InstructBLIP上实现83%攻击成功率，扰动几乎不可见。
- 攻击样本具强迁移性，适合评估大模型安全漏洞的研究者使用。
大型视觉语言模型（LVLM）在多模态任务中表现出卓越能力，但对对抗性越狱攻击仍脆弱，攻击者可施加微小扰动绕过安全机制并触发有害输出。现有白盒攻击需完全访问模型，计算成本高且泛化性差，不适用于真实黑箱场景。为此，本文提出基于零阶优化的黑箱越狱攻击方法——ZO-SPSA，其优势包括：(i) 无需模型知识，通过输入输出交互实现梯度无损近似；(ii) 无需代理模型，通用性强；(iii) 资源消耗低，显著降低显存占用。我们在InstructBLIP、LLaVA和MiniGPT-4三款模型上验证该方法，在InstructBLIP上达到83.0%最高攻击成功率，扰动程度与白盒方法相当。此外，从MiniGPT-4生成的对抗样本在其他模型上转移成功率高达64.18%，揭示了当前LVLM安全机制的关键缺陷，证明黑箱越狱在现实中的可行性。
原文摘要 · Abstract (English)
Recent advancements in Large Vision-Language Models (LVLMs) have shown groundbreaking capabilities across diverse multimodal tasks. However, these models remain vulnerable to adversarial jailbreak attacks, where adversaries craft subtle perturbations to bypass safety mechanisms and trigger harmful outputs. Existing white-box attacks methods require full model accessibility, suffer from computing costs and exhibit insufficient adversarial transferability, making them impractical for real-world, black-box settings. To address these limitations, we propose a black-box jailbreak attack on LVLMs via Zeroth-Order optimization using Simultaneous Perturbation Stochastic Approximation (ZO-SPSA). ZO-SPSA provides three key advantages: (i) gradient-free approximation by input-output interactions without requiring model knowledge, (ii) model-agnostic optimization without the surrogate model and (iii) lower resource requirements with reduced GPU memory consumption. We evaluate ZO-SPSA on three LVLMs, including InstructBLIP, LLaVA and MiniGPT-4, achieving the highest jailbreak success rate of 83.0% on InstructBLIP, while maintaining imperceptible perturbations comparable to white-box methods. Moreover, adversarial examples generated from MiniGPT-4 exhibit strong transferability to other LVLMs, with ASR reaching 64.18%. These findings underscore the real-world feasibility of black-box jailbreaks and expose critical weaknesses in the safety mechanisms of current LVLMs
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。