用强化学习训练视觉语言模型，让其更谨慎地判断内容安全。
GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning
- 通过在线强化学习激励模型先推理再决策
- 在12.3万样本上训练，平均F1比第二名高19.27%
- 适合需要高安全性的内容审核场景
为提升视觉语言模型的安全性，本文提出一种基于推理的防护模型GuardReasoner-VL。核心思想是通过在线强化学习，激励防护模型在做出内容审核决策前进行深入推理。首先构建包含12.3万样本、63.1万推理步骤的推理语料库，覆盖文本、图像及图文输入。基于此语料，通过监督微调（SFT）冷启动模型的推理能力，并进一步通过在线强化学习优化审核推理。为提升样本多样性和难度，采用拒绝采样结合提出的安全感知数据拼接进行数据增强；同时引入动态裁剪参数，早期鼓励探索，后期侧重利用。为平衡性能与生成效率，设计长度感知的安全奖励，综合考虑准确率、格式规范性和令牌成本。大量实验表明，该模型表现显著优于基线，平均F1分数领先第二名19.27%。相关数据、代码及3B/7B模型已开源。
原文摘要 · Abstract (English)
To enhance the safety of VLMs, this paper introduces a novel reasoning-based VLM guard model dubbed GuardReasoner-VL. The core idea is to incentivize the guard model to deliberatively reason before making moderation decisions via online RL. First, we construct GuardReasoner-VLTrain, a reasoning corpus with 123K samples and 631K reasoning steps, spanning text, image, and text-image inputs. Then, based on it, we cold-start our model's reasoning ability via SFT. In addition, we further enhance reasoning regarding moderation through online RL. Concretely, to enhance diversity and difficulty of samples, we conduct rejection sampling followed by data augmentation via the proposed safety-aware data concatenation. Besides, we use a dynamic clipping parameter to encourage exploration in early stages and exploitation in later stages. To balance performance and token efficiency, we design a length-aware safety reward that integrates accuracy, format, and token cost. Extensive experiments demonstrate the superiority of our model. Remarkably, it surpasses the runner-up by 19.27% F1 score on average. We release data, code, and models (3B/7B) of GuardReasoner-VL at https://github.com/yueliu1999/GuardReasoner-VL/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。