让视觉语言模型重获安全识别能力，不训练不降效
Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility
- 通过强化视觉中的危险信号，恢复模型对风险的敏感度
- 在多个攻击测试中成功率降低超过90%，任务性能无损
- 无需训练、无需修改模型，适合快速部署到现有系统
视觉语言模型（VLMs）将大语言模型（LLMs）的推理能力拓展至跨模态场景，但仍易受多模态越狱攻击。现有防御方法主要依赖安全微调或激进的标记操作，导致训练成本高或严重损害模型实用性。最新研究发现，LLMs本身具备识别文本中不安全内容的能力，而视觉输入的引入常稀释风险信号。为此，我们提出轻量级、无需训练的安全校准框架Risk Awareness Injection（RAI），通过构建语言嵌入中的不安全原型子空间，并对高风险视觉标记进行定向调制，显式激活跨模态特征空间中的安全关键信号。该机制恢复了模型从视觉输入中检测不安全内容的LLM级能力，同时保持原始标记的语义完整性以支持跨模态推理。在多个越狱攻击与实用性基准上的实验证明，RAI显著降低攻击成功率，且不牺牲任务表现。
原文摘要 · Abstract (English)
Vision language models (VLMs) extend the reasoning capabilities of large language models (LLMs) to cross-modal settings, yet remain highly vulnerable to multimodal jailbreak attacks. Existing defenses predominantly rely on safety fine-tuning or aggressive token manipulations, incurring substantial training costs or significantly degrading utility. Recent research shows that LLMs inherently recognize unsafe content in text, and the incorporation of visual inputs in VLMs frequently dilutes risk-related signals. Motivated by this, we propose Risk Awareness Injection (RAI), a lightweight and training-free framework for safety calibration that restores LLM-like risk recognition by amplifying unsafe signals in VLMs. Specifically, RAI constructs an Unsafe Prototype Subspace from language embeddings and performs targeted modulation on selected high-risk visual tokens, explicitly activating safety-critical signals within the cross-modal feature space. This modulation restores the model's LLM-like ability to detect unsafe content from visual inputs, while preserving the semantic integrity of original tokens for cross-modal reasoning. Extensive experiments across multiple jailbreak and utility benchmarks demonstrate that RAI substantially reduces attack success rate without compromising task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。