用强化学习提升视觉语言模型的推理能力，更适应真实场景变化。
Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models
- 分两阶段训练：先用思维链数据激活推理能力，再用强化学习生成多路径回答
- 在计数、结构感知等任务上达到领先水平，少样本下表现尤为出色
- 适合需要强泛化能力的视觉推理应用，如跨领域智能系统
视觉推理在理解复杂多模态数据中至关重要，推动特定领域应用与通用人工智能发展。现有方法通过精心标注的思维链（CoT）监督微调增强视觉语言模型（VLMs），但易导致过拟合与认知僵化，限制域迁移下的泛化能力。为此，我们提出Reason-RFT，一种两阶段强化微调框架。首先，使用精选的思维链数据进行监督微调（SFT），激发VLM的推理潜力；随后采用基于组相对策略优化（GRPO）的强化学习，生成多条推理-响应对，提升对域偏移的适应性。为评估该方法，我们重建了一个涵盖视觉计数、结构感知和空间变换的综合性数据集，作为三大维度的基准评估工具。实验表明：(1) 性能提升，Reason-RFT实现当前最佳结果，超越开源与闭源模型；(2) 泛化优势，在多种任务和域偏移下保持稳健性能；(3) 数据高效，在少样本学习场景中优于全量数据监督微调基线。Reason-RFT引入了视觉推理的新训练范式，是多模态研究的重要进展。项目主页：https://tanhuajie.github.io/ReasonRFT
原文摘要 · Abstract (English)
Visual reasoning abilities play a crucial role in understanding complex multimodal data, advancing both domain-specific applications and artificial general intelligence (AGI). Existing methods enhance Vision-Language Models (VLMs) through Chain-of-Thought (CoT) supervised fine-tuning using meticulously annotated data. However, this approach may lead to overfitting and cognitive rigidity, limiting the model's generalization ability under domain shifts and reducing real-world applicability. To overcome these limitations, we propose Reason-RFT, a two-stage reinforcement fine-tuning framework for visual reasoning. First, Supervised Fine-Tuning (SFT) with curated CoT data activates the reasoning potential of VLMs. This is followed by reinforcement learning based on Group Relative Policy Optimization (GRPO), which generates multiple reasoning-response pairs to enhance adaptability to domain shifts. To evaluate Reason-RFT, we reconstructed a comprehensive dataset covering visual counting, structural perception, and spatial transformation, serving as a benchmark for systematic assessment across three key dimensions. Experimental results highlight three advantages: (1) performance enhancement, with Reason-RFT achieving state-of-the-art results and outperforming both open-source and proprietary models; (2) generalization superiority, maintaining robust performance under domain shifts across various tasks; and (3) data efficiency, excelling in few-shot learning scenarios and surpassing full-dataset SFT baselines. Reason-RFT introduces a novel training paradigm for visual reasoning and marks a significant step forward in multimodal research. Project website: https://tanhuajie.github.io/ReasonRFT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。