arXiv:2502.19269cs.CV2025-02

通过优化文本提示净化CLIP模型中的后门,提升安全性和准确率。

Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in CLIP

  • 针对后门触发器设计类别的文本提示优化方法
  • 在7种攻击下实现58.83%清洁准确率与0.39%攻击成功率
  • 无需微调模型参数,适合快速部署的防御场景

尽管预训练视觉-语言模型(如CLIP)在多模态数据上表现出强大的表征能力,但近期研究揭示其易受后门攻击。现有防御策略多通过微调整个可疑模型缓解威胁,但大量参数导致优化困难,难以抵御先进攻击且常损害干净准确率。为此,我们提出类别级后门提示调优(CBPT),一种高效防御机制,通过操作文本提示间接净化被污染的CLIP模型。首先利用精心设计的正负样本进行对比学习,有效逆向提取攻击者可能使用的后门触发器;一旦建立伪触发器,再通过三个设计合理的损失函数优化类别级文本提示,调整模型决策边界,重新分类受后门影响的特征区域。大量实验表明,CBPT显著缓解后门威胁的同时保持模型实用性,例如在七种主流后门攻击下平均清洁准确率(CA)达58.83%,攻击成功率(ASR)仅为0.39%。结果凸显了提示净化设计在增强CLIP鲁棒性方面的优势。

原文摘要 · Abstract (English)

While pre-trained Vision-Language Models (VLMs) such as CLIP exhibit impressive representational capabilities for multimodal data, recent studies have revealed their vulnerability to backdoor attacks. To alleviate the threat, existing defense strategies primarily focus on fine-tuning the entire suspicious model. However, the substantial model parameters increase the difficulty of reaching a stable and consistent optimization direction, limiting their resistance against state-of-the-art attacks and often resulting in a degradation of clean accuracy. To address this challenge, we propose Class-wise Backdoor Prompt Tuning (CBPT), an efficient and effective defense mechanism that operates on text prompts to indirectly purify poisoned CLIP. Specifically, we first employ the advanced contrastive learning via carefully crafted positive and negative samples, to effectively invert the backdoor triggers that are potentially adopted by the attacker. Once the dummy trigger is established, we leverage three well-designed loss functions to optimize these class-wise text prompts, modifying the model's decision boundary and further reclassifying the feature regions affected by backdoor triggers. Extensive experiments demonstrate that CBPT significantly mitigates backdoor threats while preserving model utility, e.g. an average Clean Accuracy (CA) of 58.83% and an Attack Success Rate (ASR) of 0.39% across seven mainstream backdoor attacks. These results underscore the superiority of our prompt purifying design to strengthen CLIP's robustness against backdoor attacks.

后门防御CLIP提示调优

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。