让模型自己验证答案，用自我纠错提升多模态推理能力
SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning
- 模型自动生成答案并判断对错，错则重思，对则定稿
- 相比强基线，准确率大幅提升，且验证次数减少
- 无需外部监督，适合想提升视觉语言模型推理的开发者
我们提出自验证推理器（SVR-R1），一种多轮强化学习框架，将模型自身的验证作为多模态推理的学习信号。每个查询中，模型使用相同权重生成答案并做出二元自判断（是/否）；若为‘否’，则触发二次思考；若为‘是’或达到轮次上限，则输出最终答案并计算基于结果的奖励。SVR-R1采用GRPO与异步多轮采样框架，无需外部监督或辅助评判器。在视觉-语言推理基准上评估显示，其准确率显著优于强基线标准GRPO。训练动态表明，验证次数逐渐减少，但测试准确率上升，说明策略逐步内化自我修正能力，并通过本框架选择最自信的答案。SVR-R1填补了推理时自我精炼与强化学习训练之间较少探索的交集，为启动多模态推理提供简单而有效的方案。我们将开源SVR-R1以促进视觉语言模型的后续研究。
原文摘要 · Abstract (English)
We introduce Self-Verified Reasoner (SVR-R1), a multi-turn RL framework that turns a model's own verification into a learning signal for multimodal reasoning. For each query, the model proposes an answer using the same weights, and issues a binary self-verdict (Yes/No). A 'No' triggers a second-chance rethink; a 'Yes,' or a turn cap, finalizes the output for computing the outcome-based reward. SVR-R1 is implemented with GRPO and an asynchronous multi-turn rollout framework and needs no external supervision or auxiliary critics. We evaluate SVR-R1 on vision-language reasoning benchmarks and show that it improves accuracy by a large margin over strong standard GRPO baselines. Training dynamics show decreasing reliance on verification-fewer verification turns, yet higher test accuracy-indicating that the gap between verification and generation narrows as the policy internalizes self-correction and chooses the most confident answer via our framework. SVR-R1 bridges the less explored intersection of inference-time self-refinement and RL training for VLMs, offering a simple yet effective recipe for bootstrapping multimodal reasoning. We will open-source \textbf{SVR-R1} to facilitate future research in VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。