不训练模型，用新方法让多模态大模型更安全地生成内容。
GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models
- 用最优传输技术精准定位图像中危险区域。
- 通过注意力重分配让安全提示全程生效，减少39%的不当回答。
- 适合需要高安全性的多模态应用开发者使用。
多模态大语言模型在视觉-语言推理任务中取得显著进展，但安全性仍是关键挑战。现有输入端防御方法依赖CLIP检测不安全图像并添加安全前缀，但在复杂场景中检测不准，且解码过程中安全信号不稳定。为此，我们提出GuardAlign，一种无需训练的防御框架，融合两种策略：首先，基于最优传输的增强型安全检测利用图像块与不安全语义间的分布距离，精准识别恶意区域，无额外计算开销；其次，跨模态注意力校准通过自适应调整各层注意力权重，强化安全前缀影响，确保安全信号在生成全过程持续激活。在六个代表性多模态大模型上的实验表明，GuardAlign在SPA-VL上将不安全响应率降低39%，同时保持模型性能，在VQAv2上准确率从78.51%提升至79.21%。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) have achieved remarkable progress in vision-language reasoning tasks, yet ensuring their safety remains a critical challenge. Recent input-side defenses detect unsafe images with CLIP and prepend safety prefixes to prompts, but they still suffer from inaccurate detection in complex scenes and unstable safety signals during decoding. To address these issues, we propose GuardAlign, a training-free defense framework that integrates two strategies. First, OT-enhanced safety detection leverages optimal transport to measure distribution distances between image patches and unsafe semantics, enabling accurate identification of malicious regions without additional computational cost. Second, cross-modal attentive calibration strengthens the influence of safety prefixes by adaptively reallocating attention across layers, ensuring that safety signals remain consistently activated throughout generation. Extensive evaluations on six representative MLLMs demonstrate that GuardAlign reduces unsafe response rates by up to 39% on SPA-VL, while preserving utility, achieving an improvement on VQAv2 from 78.51% to 79.21%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。