分两阶段强化学习,提升视觉语言模型的感知与推理能力。
Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models
- 先优化视觉感知,再提升推理能力,分步训练更有效。
- 在7个基准数据集上性能显著优于基线模型。
- 适合需要强视觉理解与逻辑推理的应用场景。
强化学习(RL)在激发大语言模型(LLM)推理能力方面表现优异。受此启发,近期研究尝试将类似方法应用于视觉语言模型(VLM),以提升其推理表现。然而,直接将适用于LLM的RL方法移植到VLM存在局限,因为VLM面临的任务更为复杂:必须先准确感知和理解视觉输入,才能进行有效推理。为此,我们提出一种两阶段强化学习框架,旨在协同增强VLM的感知与推理能力。为缓解强化学习中常见的优势消失问题,我们采用数据集级采样,利用不同数据源针对性强化特定能力。训练过程中,第一阶段聚焦于粗粒度与细粒度的视觉理解,以提升感知能力;第二阶段则专注于增强推理能力。经过该两阶段强化学习流程,我们得到PeBR-R1,一个在感知与推理方面均有显著提升的视觉语言模型。在七个基准数据集上的实验结果验证了该方法的有效性,证明PeBR-R1在多样化视觉推理任务中表现出色。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has proven highly effective in eliciting the reasoning capabilities of large language models (LLMs). Inspired by this success, recent studies have explored applying similar techniques to vision-language models (VLMs), aiming to enhance their reasoning performance. However, directly transplanting RL methods from LLMs to VLMs is suboptimal, as the tasks faced by VLMs are inherently more complex. Specifically, VLMs must first accurately perceive and understand visual inputs before reasoning can be effectively performed. To address this challenge, we propose a two-stage reinforcement learning framework designed to jointly enhance both the perceptual and reasoning capabilities of VLMs. To mitigate the vanishing advantage issue commonly observed in RL training, we first perform dataset-level sampling to selectively strengthen specific capabilities using distinct data sources. During training, the first stage focuses on improving the model's visual perception through coarse- and fine-grained visual understanding, while the second stage targets the enhancement of reasoning abilities. After the proposed two-stage reinforcement learning process, we obtain PeBR-R1, a vision-language model with significantly enhanced perceptual and reasoning capabilities. Experimental results on seven benchmark datasets demonstrate the effectiveness of our approach and validate the superior performance of PeBR-R1 across diverse visual reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。