让多模态模型在推理时主动避险,提升安全性和可靠性。
SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models
- 通过强化学习将安全约束嵌入多模态推理过程,而非仅过滤输出。
- 在6个基准上安全得分78.97,优于数十倍更大的模型。
- 支持动态、可解释的安全决策,适合高风险场景应用。
多模态大推理模型虽具备强大跨模态推理能力,但在对抗性或不安全提示下易放大安全风险,这一现象称为‘推理税’。现有防御措施多作用于输出层,未约束推理过程,导致模型仍存在隐含风险。本文提出SaFeR-VLM,一种安全对齐的强化学习框架,将安全直接嵌入多模态推理。框架包含四部分:(I) QI-Safe-10K数据集,聚焦安全关键与推理敏感案例;(II) 安全感知回溯,对不安全生成进行反思与修正而非丢弃;(III) 多维加权结构化奖励建模,对幻觉和矛盾施加显式惩罚;(IV) GRPO优化,强化安全且修正后的轨迹。该设计使安全从被动防护变为推理主动驱动力,实现可扩展、泛化的安全推理。SaFeR-VLM-3B在六项基准上平均安全得分70.13、助益得分78.97,超越同规模及超十倍大的模型(如Skywork-R1V3-38B、Qwen2.5VL-72B、GLM4.5V-106B)。SaFeR-VLM-7B因规模提升,安全得分分别超过GPT-5-mini和Gemini-2.5-Flash达6.47和16.76点,且助益性能无下降。代码已开源。
原文摘要 · Abstract (English)
Multimodal Large Reasoning Models (MLRMs) demonstrate impressive cross-modal reasoning but often amplify safety risks under adversarial or unsafe prompts, a phenomenon we call the \textit{Reasoning Tax}. Existing defenses mainly act at the output level and do not constrain the reasoning process, leaving models exposed to implicit risks. In this paper, we propose SaFeR-VLM, a safety-aligned reinforcement learning framework that embeds safety directly into multimodal reasoning. The framework integrates four components: (I) QI-Safe-10K, a curated dataset emphasizing safety-critical and reasoning-sensitive cases; (II) safety-aware rollout, where unsafe generations undergo reflection and correction instead of being discarded; (III) structured reward modeling with multi-dimensional weighted criteria and explicit penalties for hallucinations and contradictions; and (IV) GRPO optimization, which reinforces both safe and corrected trajectories. This unified design shifts safety from a passive safeguard to an active driver of reasoning, enabling scalable and generalizable safety-aware reasoning. SaFeR-VLM further demonstrates robustness against both explicit and implicit risks, supporting dynamic and interpretable safety decisions beyond surface-level filtering. SaFeR-VLM-3B achieves average performance $70.13$ and $78.97$ on safety and helpfulness across six benchmarks, surpassing both same-scale and $>10\times$ larger models such as Skywork-R1V3-38B, Qwen2.5VL-72B, and GLM4.5V-106B. Remarkably, SaFeR-VLM-7B benefits from its increased scale to surpass GPT-5-mini and Gemini-2.5-Flash by \num{6.47} and \num{16.76} points respectively on safety metrics, achieving this improvement without any degradation in helpfulness performance. Our codes are available at https://github.com/HarveyYi/SaFeR-VLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。