构建多语言多模态安全评测基准,揭示模型在不同语言和模态下的安全短板。
Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models
- 设计10万条跨语言跨模态有害图文对,区分图像主导与文本主导风险源。
- 发现非高资源语言和非拉丁文字在文本主导时威胁更强,且模型存在语言-模态不对称脆弱性。
- 指出单纯扩大模型规模无法解决安全不平等问题,需针对性对齐策略。
视觉-语言大模型(VLLMs)在多语言与多模态联合威胁下的安全性仍严重缺乏研究。现有评测基准通常只覆盖单维度:或为多语言但仅限文本,或为多模态但仅限单语言。近期红队测试虽尝试结合图文,但依赖字体风格图像且图文语义关联不足,难以反映真实跨模态交互。为此,我们提出Lingua-SafetyBench,包含100,440条跨越10种语言的有害图文对。关键在于将数据划分为图像主导与文本主导子集,精准分离风险来源。大量评估显示,当前VLLMs在联合输入下仍存在显著漏洞:非高资源语言(Non-HRLs)及非拉丁脚本通常构成更大威胁;模态-语言交互分析揭示显著不对称性——在高资源语言(HRLs)中,图像主导风险更致命;而在Non-HRLs中,文本主导风险严重削弱安全表现。对Qwen系列的控制实验表明,模型规模扩展与迭代升级虽整体提升安全性,但对HRLs收益更大,加剧了HRLs与Non-HRLs间在文本主导风险上的安全差距。这说明实现鲁棒安全需超越单纯缩放,采用语言与模态感知的专门对齐策略。代码与数据集将在https://github.com/zsxr15/Lingua-SafetyBench公开。警告:本文含不安全内容示例。
原文摘要 · Abstract (English)
The robust safety of Vision-Language Large Models (VLLMs) against joint multilingual and multimodal threats remains severely underexplored. Current benchmarks typically isolate these dimensions, being either multilingual but text-only, or multimodal but monolingual. While recent red-teaming efforts attempt to bridge this gap by rendering harmful prompts as images, their overreliance on typography-style visuals and lack of semantically grounded image-text pairs fail to capture realistic cross-modal interactions under multilingual and multimodal conditions. To address this, we introduce Lingua-SafetyBench, a comprehensive benchmark of 100,440 harmful image-text pairs spanning 10 languages. Crucially, Lingua-SafetyBench explicitly partitions data into image-dominant and text-dominant subsets to precisely disentangle sources of risk. Extensive evaluations reveal that current VLLMs retain non-negligible vulnerabilities under these joint inputs. Linguistically, requests in Non-High-Resource Languages (Non-HRLs) and non-Latin scripts generally pose greater threats. Furthermore, analyzing modality-language interactions uncovers a striking asymmetry: in High-Resource Languages (HRLs), models are most vulnerable to image-dominant risks, whereas in Non-HRLs, text-dominant risks severely degrade safety performance. Finally, a controlled study on the Qwen series demonstrates that while model scaling and iterative upgrades improve overall safety, they disproportionately benefit HRLs. This exacerbates the safety disparity between HRLs and Non-HRLs under text-dominant risks, highlighting that achieving robust safety requires dedicated language- and modality-aware alignment strategies beyond mere scaling. The code and dataset will be available at https://github.com/zsxr15/Lingua-SafetyBench.Warning: this paper contains examples with unsafe content.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。