通过忽略功能词提升视觉语言模型抗攻击能力
Pay Less Attention to Function Words for Free Robustness of Vision-Language Models
- 设计差分注意力机制，分离并抑制功能词的干扰
- 在检索任务上平均降低18%攻击成功率，性能仅降0.2%
- 无需额外训练，可通用适配多种模型与任务
为解决鲁棒VLM中鲁棒性与性能的权衡问题，我们发现功能词会引发视觉语言模型对跨模态对抗攻击的脆弱性，提出功能词去注意力（FDA）来缓解此问题。类似差分放大器，FDA在注意力头内计算原始与功能词的交叉注意力，并差分相减，以实现更对齐、更鲁棒的模型。实验涵盖2个SOTA基线、6种攻击、3个模型、2个下游任务和3个数据集。整体上，FDA在检索任务中实现平均18%/13%/53%的攻击成功率下降，性能仅损失0.2%/0.3%/0.6%；在视觉定位任务中实现90%攻击成功率下降，且性能提升0.3%。实验验证了FDA的可扩展性、泛化性和零样本性能，并进行深入消融分析。代码已开源。
原文摘要 · Abstract (English)
To address the trade-off between robustness and performance for robust VLM, we observe that function words could incur vulnerability of VLMs against cross-modal adversarial attacks, and propose Function-word De-Attention (FDA) accordingly to mitigate the impact of function words. Similar to differential amplifiers, our FDA calculates the original and the function-word cross-attention within attention heads, and differentially subtracts the latter from the former for more aligned and robust VLMs. Comprehensive experiments include 2 SOTA baselines under 6 different attacks on 2 downstream tasks, 3 datasets, and 3 models. Overall, our FDA yields an average 18/13/53% ASR drop with only 0.2/0.3/0.6% performance drops on the 3 tested models on retrieval, and a 90% ASR drop with a 0.3% performance gain on visual grounding. We demonstrate the scalability, generalization, and zero-shot performance of FDA experimentally, as well as in-depth ablation studies and analysis. Code is available at https://github.com/michaeltian108/FDA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。