用统计方法提升多模态后门检测的可靠性,避免误判。
When Modalities Fail to Tango: Conformal Backdoor Detection in Multimodal Contrastive Learning

- 引入置信区间框架,通过非相似性分数识别异常对
- 在CC3M数据集上实现100%召回率下5.79%假阳性率
- 适合关注多模态模型安全性的研究人员
多模态对比学习(MCL)中的后门攻击近年来受到广泛关注,因许多下游任务依赖预训练的MCL模型。现有基于检测的防御方法主要依赖CLIPScore指标,假设中毒图像-文本对的语义相似度较低。然而我们发现两种关键缺陷:(1) 良性和中毒对的CLIPScore分布存在显著重叠,削弱了该指标的可靠性;(2) 固定阈值检测无法为重叠区域中的模糊样本提供统计保证。为此,我们提出整合置信预测(CP)框架,利用非相似性分数(NCS)量化不确定性,建立可证明的置信区间以检测中毒对。在此基础上,我们提出CASCADE——一种两级粗到细的反向门检测框架。粗粒度阶段利用跨模态一致性识别高置信度良性与中毒对;细粒度阶段从高置信度中毒对构建参考集,计算每个未识别样本在文本空间的实例级NCS,衡量其与中毒分布的一致性,从而精准识别未识别子集中潜在的中毒对。在大规模CC3M数据集上的大量实验表明,CASCADE在多种攻击下平均达到100%真阳性率、5.79%假阳性率,平均AUROC达0.9867,且对自适应攻击仍有效。
原文摘要 · Abstract (English)
Backdoor attacks in multimodal contrastive learning (MCL) have garnered growing attention in recent years, as many downstream tasks critically depend on pre-trained MCL models. Existing detection-based defenses predominantly rely on the CLIPScore metric, under the assumption that poisoned pairs exhibit lower semantic similarity between the image and the caption. However, we identify two critical flaws remaining in existing methods: (1) the substantial overlap between CLIPScore distributions of benign and poisoned pairs undermines the reliability of this metric, and (2) fixed-threshold detection cannot provide statistical guarantees for ambiguous samples within overlapping regions. To overcome these limitations, we propose integrating conformal prediction (CP), a statistical framework that quantifies uncertainty through nonconformity scores (NCSs), to establish provable confidence bounds for detecting poisoned image-caption pairs. Building on CP, we introduce CASCADE, a novel two-stage Coarse-to-Fine Conformal Backdoor Detection framework. The coarse-grained stage uses cross-modality consistency to identify high-confidence benign and poisoned pairs. In the fine-grained stage, a reference set is constructed from high-confidence poisoned pairs, and instance-level NCSs based on text-space similarity are computed for each sample in the unidentified subset. These NCSs measure conformity to the poisoning distribution and enable precise identification of latent poisoned pairs within the unidentified subset. Extensive experiments on the large-scale CC3M dataset demonstrate that CASCADE achieves an average FPR of 5.79% at 100% TPR and an average AUROC of 0.9867 across diverse attacks, while remaining effective against adaptive attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。