提出轻量级防御框架，高效识别视觉语言模型中的恶意提示攻击。
VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Prompts
- 通过多模态特征提取发现良性与恶意提示的分布差异
- 在多个数据集上实现高鲁棒性与低延迟检测
- 可直接嵌入现有系统，适合安全部署场景
视觉语言模型（VLMs）在视觉融合过程中对齐能力减弱，易受恶意提示攻击。现有防御方法在效率和鲁棒性上存在不足。本文提出多模态聚合特征提取（MAFE）框架，使CLIP能够处理长文本，并将多模态信息融合为统一表征。通过对MAFE提取特征的实证分析，发现良性与恶意提示存在显著分布差异。基于此，我们构建了VLMShield——一种轻量级安全检测器，可作为即插即用方案高效识别多模态恶意攻击。大量实验表明，该方法在鲁棒性、效率和实用性方面均表现优异。代码已开源。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) face significant safety vulnerabilities from malicious prompt attacks due to weakened alignment during visual integration. Existing defenses suffer from efficiency and robustness. To address these challenges, we first propose the Multimodal Aggregated Feature Extraction (MAFE) framework that enables CLIP to handle long text and fuse multimodal information into unified representations. Through empirical analysis of MAFE-extracted features, we discover distinct distributional patterns between benign and malicious prompts. Building upon this finding, we develop VLMShield, a lightweight safety detector that efficiently identifies multimodal malicious attacks as a plug-and-play solution. Extensive experiments demonstrate superior performance across multiple dimensions, including robustness, efficiency, and utility. Through our work, we hope to pave the way for more secure multimodal AI deployment. Code is available at [this https URL](https://github.com/pgqihere/VLMShield).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。