通过双阶段对抗训练提升视觉语言模型抗干扰能力。
Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
- 从头开始用大规模对抗数据预训练，构建基础防御模型
- ΔCLIP在ImageNet-1k上抗扰动能力提升20%，Δ²LLaVA任务性能增30%
- 适合关注模型安全与鲁棒性的研究者和工程师
本文研究视觉语言模型在对抗性视觉扰动下的鲁棒性，提出一种新型的“双视觉防御”机制。不同于以往对预训练CLIP进行轻量级对抗微调的方法，本文从零开始使用网络规模数据进行大规模对抗视觉-语言预训练。随后通过引入对抗性视觉指令微调进一步强化防御能力。经两阶段训练得到的ΔCLIP与Δ²LLaVA模型，在零样本鲁棒性方面显著增强，并在视觉语言模型对抗防御任务中达到新基准。例如，ΔCLIP在ImageNet-1k上的抗扰动能力较之前最优模型提升约20%；Δ²LLaVA相较先前方法，在图像描述任务中鲁棒性提升约30%，在视觉问答任务中提升约20%。此外，该模型还展现出更强的零样本识别能力、更少幻觉现象以及更优推理表现。
原文摘要 · Abstract (English)
This paper investigates the robustness of vision-language models against adversarial visual perturbations and introduces a novel ``double visual defense" to enhance this robustness. Unlike previous approaches that resort to lightweight adversarial fine-tuning of a pre-trained CLIP model, we perform large-scale adversarial vision-language pre-training from scratch using web-scale data. We then strengthen the defense by incorporating adversarial visual instruction tuning. The resulting models from each stage, $Δ$CLIP and $Δ^2$LLaVA, show substantially enhanced zero-shot robustness and set a new state-of-the-art in adversarial defense for vision-language models. For example, the adversarial robustness of $Δ$CLIP surpasses that of the previous best models on ImageNet-1k by ~20%. %For example, $Δ$CLIP surpasses the previous best models on ImageNet-1k by ~20% in terms of adversarial robustness. Similarly, compared to prior art, $Δ^2$LLaVA brings a ~30% robustness improvement to image captioning task and a ~20% robustness improvement to visual question answering task. Furthermore, our models exhibit stronger zero-shot recognition capability, fewer hallucinations, and superior reasoning performance compared to baselines. Our project page is https://doublevisualdefense.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。