针对多模态模型后门攻击，提出高效定位与修复方法
Robust Defense Strategies for Multimodal Contrastive Learning: Efficient Fine-tuning Against Backdoor Attacks
- 利用图像分割作为监督信号识别后门触发器
- 精准定位受污染标签和样本，构建小规模修复数据集
- 无需全量重训，适用于实际部署的模型加固
多模态深度学习模型（如CLIP）在图像-文本理解等任务中表现优异，但易受后门攻击影响，攻击者可隐秘操控模型行为。现有防御方法通常需从头训练或使用大规模数据微调，且无法精确定位受攻击的标签。本文提出一种新型防御策略：针对被污染的CLIP模型，通过引入图像分割“预言机”作为输出监督，开发两种算法——一是区分CLIP与预言机的知识差异以识别潜在触发器；二是精准定位受影响的标签和样本，并构建紧凑的微调数据集。基于此知识，可高效修复被污染模型，消除后门效应。在多个视觉识别基准上的实验表明，该方法在基于CLIP的后门防御中具有显著有效性。
原文摘要 · Abstract (English)
The advent of multimodal deep learning models, such as CLIP, has unlocked new frontiers in a wide range of applications, from image-text understanding to classification tasks. However, these models are not safe for adversarial attacks, particularly backdoor attacks, which can subtly manipulate model behavior. Moreover, existing defense methods typically involve training from scratch or fine-tuning using a large dataset without pinpointing the specific labels that are affected. In this study, we introduce an innovative strategy to enhance the robustness of multimodal contrastive learning models against such attacks. In particular, given a poisoned CLIP model, our approach can identify the backdoor trigger and pinpoint the victim samples and labels in an efficient manner. To that end, an image segmentation ``oracle'' is introduced as the supervisor for the output of the poisoned CLIP. We develop two algorithms to rectify the poisoned model: (1) differentiating between CLIP and Oracle's knowledge to identify potential triggers; (2) pinpointing affected labels and victim samples, and curating a compact fine-tuning dataset. With this knowledge, we are allowed to rectify the poisoned CLIP model to negate backdoor effects. Extensive experiments on visual recognition benchmarks demonstrate our strategy is effective in CLIP-based backdoor defense.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。