用噪声对抗噪声，无需训练即可净化视觉对抗样本。
Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs
- 引入随机扰动的对抗样本作为参考，利用跨模态注意力修复模型输出。
- 在多个数据集上实现显著净化效果，且无需额外训练。
- 无需训练、计算高效，适合工业级大规模部署。
大型视觉语言模型（LVLMs）在多模态任务中展现出强大能力，但易受视觉对抗攻击影响。本文提出F3，一种创新的对抗净化框架，采用‘以火攻火’策略：通过向对抗样本注入简单扰动，反向抑制其危害。F3利用随机扰动后对抗样本的跨模态注意力作为参考目标，通过加噪优化其注意力分布，从而生成更清晰可靠的输出。该看似矛盾的方法取得了显著净化效果。此外，F3无需训练、实现简单，相比现有方法具有显著计算效率优势，适用于对鲁棒性与效率要求高的大规模工业场景。代码已开源。
原文摘要 · Abstract (English)
Recent advances in large vision-language models (LVLMs) have showcased their remarkable capabilities across a wide range of multimodal vision-language tasks. However, these models remain vulnerable to visual adversarial attacks, which can substantially compromise their performance. In this paper, we introduce F3, a novel adversarial purification framework that employs a counterintuitive ``fighting fire with fire'' strategy: intentionally introducing simple perturbations to adversarial examples to mitigate their harmful effects. Specifically, F3 leverages cross-modal attentions derived from randomly perturbed adversary examples as reference targets. By injecting noise into these adversarial examples, F3 effectively refines their attention, resulting in cleaner and more reliable model outputs. Remarkably, this seemingly paradoxical approach of employing noise to counteract adversarial attacks yields impressive purification results. Furthermore, F3 offers several distinct advantages: it is training-free and straightforward to implement, and exhibits significant computational efficiency improvements compared to existing purification methods. These attributes render F3 particularly suitable for large-scale industrial applications where both robust performance and operational efficiency are critical priorities. The code is available at https://github.com/btzyd/F3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。