用大模型生成欺骗性文本，测试CLIP等多模态模型的组合漏洞。
Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updates
- 利用大模型生成误导性文本，攻击多模态模型的组合理解能力。
- 在图像、视频、音频上均成功暴露模型弱点，攻击成功率显著提升。
- 方法适用于小模型，适合安全评估与模型鲁棒性研究者使用。
尽管预训练多模态表示（如CLIP）表现出色，但其存在显著的组合脆弱性，导致反直觉判断。本文提出多模态对抗组合性（MAC）基准，利用大语言模型（LLMs）生成欺骗性文本样本，以探测不同模态下的漏洞，并通过逐样本攻击成功率和组级熵基多样性评估效果。为提升零样本方法性能，提出一种自训练策略，结合拒绝采样微调与促进多样性的过滤机制，有效提升攻击成功率与样本多样性。即使使用较小的语言模型（如Llama-3.1-8B），该方法也能在多种多模态表示（包括图像、视频、音频）中更有效地揭示组合性漏洞。
原文摘要 · Abstract (English)
While pre-trained multimodal representations (e.g., CLIP) have shown impressive capabilities, they exhibit significant compositional vulnerabilities leading to counterintuitive judgments. We introduce Multimodal Adversarial Compositionality (MAC), a benchmark that leverages large language models (LLMs) to generate deceptive text samples to exploit these vulnerabilities across different modalities and evaluates them through both sample-wise attack success rate and group-wise entropy-based diversity. To improve zero-shot methods, we propose a self-training approach that leverages rejection-sampling fine-tuning with diversity-promoting filtering, which enhances both attack success rate and sample diversity. Using smaller language models like Llama-3.1-8B, our approach demonstrates superior performance in revealing compositional vulnerabilities across various multimodal representations, including images, videos, and audios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。