通过反思机制提升视觉语言模型的安全性,防止生成有害内容。
Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Models
- 三阶段框架:思考-反思-修正,利用首次推理中的漏洞信号进行自我修正。
- 在Qwen2.5-VL-7B上将安全响应率从42.8%提升至87.7%,显著增强抗攻击能力。
- 适用于需要高安全性的多模态应用,如医疗、金融等敏感场景。
随着多模态推理能力的提升,大型视觉语言模型(LVLMs)的安全性日益受到关注。现有方法虽能在生成前分析潜在风险,但单次推理仍易受上下文或视觉越狱攻击影响,难以发现自身输出中的有害内容。本文提出Think-Reflect-Revise(TRR)框架,通过政策引导的自我反思机制,充分利用首次推理中暴露的恶意信号实现真正自纠。构建包含5000个样本的ReSafe数据集,采用监督微调与强化学习相结合的方式训练模型。实验表明,TRR使Qwen2.5-VL-7B在安全基准和越狱攻击测试中整体安全响应率从42.8%提升至87.7%,同时保持在MMMU和MMStar等通用评测上的稳定表现。
原文摘要 · Abstract (English)
As multimodal reasoning improves the overall capabilities of Large Vision Language Models (LVLMs), recent studies have begun to explore safety-oriented reasoning, aiming to enhance safety awareness by analyzing potential safety risks during the reasoning process before generating the final response. Although such approaches improve safety awareness and interpretability, this single-pass think-then-answer paradigm remains vulnerable to contextual or visual jailbreak attacks. This reveals a critical flaw: single-pass reasoning may overlook explicit harmful content in its own output. Our key insight is to exploit this wasted signal through reflection, which can effectively leverage the malicious content revealed in the first-pass reasoning to enable genuine self-correction and prevent unsafe generations. Motivated by this, we propose Think-Reflect-Revise (TRR), a three-stage training framework designed to enhance the safety alignment of LVLMs through policy-guided self-reflection. We first build a Reflective Safety Reasoning (ReSafe) dataset with 5,000 examples that follow a think-reflect-revise process. We then fine-tune the target model using the ReSafe dataset to initialize reflective behavior, and finally reinforce policy-guided reflection through reinforcement learning. Experimental results show that TRR substantially improves the safety performance of LVLMs across both safety-awareness benchmarks and jailbreak attack evaluations, increasing the overall safe response rate from 42.8% to 87.7% on Qwen2.5-VL-7B, while preserving stable performance on general benchmarks such as MMMU and MMStar. The project page is available at https://think-reflect-revise.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。