利用视觉语言模型的内在对齐关系，零样本检测有害图像。
Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMs
- 通过投影CLIP的废弃令牌到文本空间匹配毒性描述，实现零样本检测。
- 在不修改模型的前提下，防御成功率66.9%，误报率仅3.2%。
- 轻量级方案适合部署于推理与微调阶段，适配资源受限场景。
大型视觉语言模型（LVLMs）得益于大规模视觉数据的预训练和微调，在多模态理解方面取得了显著进展。然而，尽管其文本安全机制较强，仍易受有害视觉输入影响。现有防护手段通常依赖预过滤或微调，成本高且降低整体可用性。为此，我们提出SafeCLIP，一种轻量级方法，利用LVLM固有的多模态对齐能力实现零样本有毒图像检测。通过将CLIP中被丢弃的CLS token投影至文本空间，并与毒性描述进行匹配，SafeCLIP无需任何架构改动，仅引入3.2%的延迟，即可在推理与微调阶段动态实现安全校正。实验表明，该方法达到66.9%的防御成功率，误报率仅为3.2%，开销7.2%；相比之下，当前最优方法仅达52.9%成功率，误报率达10.7%，开销高达210%。本工作证明，利用内在多模态对齐可实现高效、低成本的LVLM安全防护。代码已公开于anonymous.4open.science/r/safeclip-2C01。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have made significant strides in multimodal comprehension, thanks to extensive pre-training and fine-tuning on large-scale visual datasets. However, despite their robust textual safety mechanisms, they remain vulnerable to harmful visual inputs. Existing safeguards-typically relying on pre-filtering or fine-tuning-incur high costs and diminish overall utility. To address this critical vulnerability, we introduce SafeCLIP, a lightweight method that leverages LVLMs inherent multimodal alignment for zero-shot toxic image detection. By projecting CLIPs discarded CLS token into its text space and matching it with toxic descriptors, SafeCLIP detects harmful content without any architectural changes-adding minimal latency and enabling dynamic safety corrections during inference and fine-tuning.Experiments show that SafeCLIP achieves a 66.9% defense success rate with only 3.2% false positive rate and 7.2% overhead. In contrast, state-of-the-art methods achieve 52.9% success but have a 10.7% false positive rate and 210% overhead. Our work demonstrates that leveraging inherent multimodal alignment can yield efficient, low-cost LVLM safety. Code is available at anonymous.4open.science/r/safeclip-2C01.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。