发现多模态安全评估中图像风险信息泄露，构建新基准测试模型真实安全性。
VLSBench: Unveiling Visual Leakage in Multimodal Safety
- 通过自动化流程构建2.2千组无泄露图文对，避免文本暴露图像风险。
- 现有模型在新基准上表现显著下降，证明旧评估存在严重偏差。
- 揭示文本对齐足以应对含泄露场景，多模态对齐更适合无泄露场景。
多模态大语言模型（MLLMs）的安全性问题日益突出。令人意外的是，以往研究发现仅用文本去偏就能达到与图文对齐相当的安全性能。为解释此现象，我们发现现有多模态安全基准存在视觉安全信息泄露（VSIL）问题：图像中的敏感内容已在文本查询中被揭示。因此，MLLMs仅凭文本即可拒绝敏感图文对，导致跨模态安全评估不可靠。我们进一步对比了文本对齐与多模态对齐的差异。为此，我们构建了多模态无泄露安全基准VLSBench，包含2.2k张无泄露的图像-文本对，通过自动化数据管道生成。实验表明，VLSBench对开源与闭源模型（如LLaVA、Qwen2-VL、GPT-4o）构成显著挑战。我们在该基准上实证比较发现，在存在VSIL时，文本对齐已足够有效；而在无泄露场景下，多模态对齐更优。代码与数据已公开于https://github.com/AI45Lab/VLSBench。
原文摘要 · Abstract (English)
Safety concerns of Multimodal large language models (MLLMs) have gradually become an important problem in various applications. Surprisingly, previous works indicate a counterintuitive phenomenon that using textual unlearning to align MLLMs achieves comparable safety performances with MLLMs aligned with image text pairs. To explain such a phenomenon, we discover a Visual Safety Information Leakage (VSIL) problem in existing multimodal safety benchmarks, i.e., the potentially risky content in the image has been revealed in the textual query. Thus, MLLMs can easily refuse these sensitive image-text pairs according to textual queries only, leading to unreliable cross-modality safety evaluation of MLLMs. We also conduct a further comparison experiment between textual alignment and multimodal alignment to highlight this drawback. To this end, we construct multimodal Visual Leakless Safety Bench (VLSBench) with 2.2k image-text pairs through an automated data pipeline. Experimental results indicate that VLSBench poses a significant challenge to both open-source and close-source MLLMs, e.g., LLaVA, Qwen2-VL and GPT-4o. Besides, we empirically compare textual and multimodal alignment methods on VLSBench and find that textual alignment is effective enough for multimodal safety scenarios with VSIL, while multimodal alignment is preferable for safety scenarios without VSIL. Code and data are released under https://github.com/AI45Lab/VLSBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。