用扩散模型生成难检测的欺骗性图文样本，实现对视觉语言模型的隐蔽后门攻击。
CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models
- 利用扩散模型得分匹配生成带触发特征的自然图像，不破坏原图语义。
- 在MSCOCO和VQA v2上实现超80%后门激活率，模型正常功能不受影响。
- 通过重要区域掩码和多模态引导提升隐蔽性，适合研究模型安全性的读者。
视觉语言模型（VLM）在图像描述和视觉问答等任务中取得显著成果，但其广泛应用也暴露了安全漏洞。现有针对VLM的后门攻击多依赖数据投毒，通过添加视觉触发器并修改文本标签，导致图像-文本不一致，易被检测。为此，本文提出基于扩散模型的无标签后门攻击方法CBV，通过在扩散模型反向生成过程中调整得分，引导生成包含触发特征的自然中毒样本。同时，利用触发图像的文本信息作为多模态引导增强攻击效果，并引入基于GradCAM的掩码机制（GM），仅在语义关键区域进行修改，提升隐蔽性。在MSCOCO和VQA v2数据集上，针对四种代表性VLM的实验表明，该方法实现超过80%的攻击成功率，同时保持模型正常功能。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have achieved remarkable success in tasks such as image captioning and visual question answering (VQA). However, as their applications become increasingly widespread, recent studies have revealed that VLMs are vulnerable to backdoor attacks. Existing backdoor attacks on VLMs primarily rely on data poisoning by adding visual triggers and modifying text labels, where the induced image-text mismatch makes poisoned samples easy to detect. To address this limitation, we propose the Clean-Label Backdoor Attack on VLMs via Diffusion Models (CBV), which leverages diffusion models to generate natural poisoned examples via score matching. Specifically, CBV modifies the score during the reverse generation process of the diffusion model to guide the generation of poisoned samples that contain triggered image features. To further enhance the effectiveness of the attack, we incorporate the textual information of the triggered images as multimodal guidance during generation. Moreover, to enhance stealthiness, we introduce a GradCAM-guided Mask (GM) that restricts modifications to only the most semantically important regions, rather than the entire image. We evaluate our method on MSCOCO and VQA v2 with four representative VLMs, achieving over 80% ASR while preserving normal functionality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。