用规则奖励提升视觉语言模型的推理能力，效果优于传统微调。
VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
- 基于视觉任务的确定性答案设计规则奖励机制，实现稳定强化学习。
- 在多个视觉理解任务上表现媲美监督微调，泛化能力更强。
- 揭示了目标检测中的奖励欺骗、模型突破时刻等关键现象。
近期DeepSeek R1表明，通过简单有效的设计，强化学习（RL）可显著提升大语言模型的推理能力。其核心在于基于规则的奖励机制，利用具有确定性真实答案的任务实现精确且稳定的奖励计算。在视觉领域，我们观察到大量视觉理解任务本身具备明确的标注真值，这一特性使其天然适配规则奖励机制。受此启发，我们探索将R1风格的强化学习扩展至视觉语言模型（VLMs），旨在增强其视觉推理能力。为此，我们提出VLM-R1框架，专门用于通过强化学习提升VLM在通用视觉语言任务上的表现。实验结果表明，基于RL的模型不仅在视觉理解任务上表现优异，且在泛化能力上超越监督微调（SFT）。我们还进行了全面消融研究，揭示了一系列重要发现：目标检测中的奖励欺骗现象、'OD aha moment'的出现、训练数据质量的影响，以及不同模型规模下强化学习的缩放行为。这些分析有助于深入理解强化学习如何提升视觉语言模型能力，我们希望研究成果与开源贡献能推动该领域持续发展。代码与模型已公开于https://github.com/om-ai-lab/VLM-R1。
原文摘要 · Abstract (English)
Recently DeepSeek R1 has shown that reinforcement learning (RL) can substantially improve the reasoning capabilities of Large Language Models (LLMs) through a simple yet effective design. The core of R1 lies in its rule-based reward formulation, which leverages tasks with deterministic ground-truth answers to enable precise and stable reward computation. In the visual domain, we similarly observe that a wide range of visual understanding tasks are inherently equipped with well-defined ground-truth annotations. This property makes them naturally compatible with rule-based reward mechanisms. Motivated by this observation, we investigate the extension of R1-style reinforcement learning to Vision-Language Models (VLMs), aiming to enhance their visual reasoning capabilities. To this end, we develop VLM-R1, a dedicated framework designed to harness RL for improving VLMs' performance on general vision-language tasks. Using this framework, we further explore the feasibility of applying RL to visual domain. Experimental results indicate that the RL-based model not only delivers competitive performance on visual understanding tasks but also surpasses Supervised Fine-Tuning (SFT) in generalization ability. Furthermore, we conduct comprehensive ablation studies that uncover a series of noteworthy insights, including the presence of reward hacking in object detection, the emergence of the "OD aha moment", the impact of training data quality, and the scaling behavior of RL across different model sizes. Through these analyses, we aim to deepen the understanding of how reinforcement learning enhances the capabilities of vision-language models, and we hope our findings and open-source contributions will support continued progress in the vision-language RL community. Our code and model are available at https://github.com/om-ai-lab/VLM-R1
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。