提出BlueSuffix防御视觉语言模型越狱攻击，兼顾安全与性能。
BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks
- 通过视觉/文本净化器和强化微调生成后缀，提升跨模态鲁棒性。
- 在4个模型、4个基准上显著优于基线，对良性输入无性能损失。
- 适合关注多模态模型安全的开发者与研究人员使用。
本文研究视觉语言模型（VLM）在黑盒设置下的越狱攻击防御。现有方法或仅增强视觉或语言模块（单模态），或通过文本-图像表征重对齐提升鲁棒性（双模态），但均存在两大局限：一是未能充分挖掘跨模态信息，二是降低模型在正常输入上的性能。为此，我们提出新型蓝队方法BlueSuffix，可在不损害模型性能的前提下有效防御越狱攻击。该方法包含三个核心组件：1）针对越狱图像的视觉净化器；2）针对越狱文本的文本净化器；3）基于强化微调的蓝队后缀生成器，用于增强跨模态鲁棒性。我们在四个VLM（LLaVA、MiniGPT-4、InstructionBLIP、Gemini）和四个安全基准（Harmful Instruction、AdvBench、MM-SafetyBench、RedTeam-2K）上进行了实证，结果表明BlueSuffix显著优于基线防御方法。代码已开源。
原文摘要 · Abstract (English)
In this paper, we focus on black-box defense for VLMs against jailbreak attacks. Existing black-box defense methods are either unimodal or bimodal. Unimodal methods enhance either the vision or language module of the VLM, while bimodal methods robustify the model through text-image representation realignment. However, these methods suffer from two limitations: 1) they fail to fully exploit the cross-modal information, or 2) they degrade the model performance on benign inputs. To address these limitations, we propose a novel blue-team method BlueSuffix that defends target VLMs against jailbreak attacks without compromising its performance under black-box setting. BlueSuffix includes three key components: 1) a visual purifier against jailbreak images, 2) a textual purifier against jailbreak texts, and 3) a blue-team suffix generator using reinforcement fine-tuning for enhancing cross-modal robustness. We empirically show on four VLMs (LLaVA, MiniGPT-4, InstructionBLIP, and Gemini) and four safety benchmarks (Harmful Instruction, AdvBench, MM-SafetyBench, and RedTeam-2K) that BlueSuffix outperforms the baseline defenses by a significant margin. Our BlueSuffix opens up a promising direction for defending VLMs against jailbreak attacks. Code is available at https://github.com/Vinsonzyh/BlueSuffix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。