用简单微调法清除CLIP模型中的后门攻击，还能保持性能。
Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP
- 通过扰动再恢复机制，仅用微调即可清除后门。
- 在多种攻击类型下，清除率高且标准任务性能不变。
- 只需合成数据，无需真实训练集，适合实际部署。
视觉-语言模型如CLIP在图像-文本关联任务中表现出色，具备强大的检索与零样本分类能力。由于其广泛使用且基于网络抓取的图文对训练，易受后门攻击。鉴于从头训练CLIP成本高昂，本文聚焦于通过微调修复被污染模型。实验表明，现有清理方法对混合型或BadNet类结构化触发器无效，暴露了现实部署中的关键漏洞。为此，提出PAR（Perturb and Recover）——一种简单但高效的后门移除机制。在不同编码器和攻击类型上验证，PAR能有效清除后门同时保持良好标准性能。进一步证明，该方法仅需合成图文对即可生效，无需真实训练数据。代码与模型已开源。
原文摘要 · Abstract (English)
Vision-Language models like CLIP have been shown to be highly effective at linking visual perception and natural language understanding, enabling sophisticated image-text capabilities, including strong retrieval and zero-shot classification performance. Their widespread use, as well as the fact that CLIP models are trained on image-text pairs from the web, make them both a worthwhile and relatively easy target for backdoor attacks. As training foundational models, such as CLIP, from scratch is very expensive, this paper focuses on cleaning potentially poisoned models via fine-tuning. We first show that existing cleaning techniques are not effective against simple structured triggers used in Blended or BadNet backdoor attacks, exposing a critical vulnerability for potential real-world deployment of these models. Then, we introduce PAR, Perturb and Recover, a surprisingly simple yet effective mechanism to remove backdoors from CLIP models. Through extensive experiments across different encoders and types of backdoor attacks, we show that PAR achieves high backdoor removal rate while preserving good standard performance. Finally, we illustrate that our approach is effective even only with synthetic text-image pairs, i.e. without access to real training data. The code and models are available on \href{https://github.com/nmndeep/PerturbAndRecover}{GitHub}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。