让视觉语言模型像人一样用图像思考，通过强化学习学会边画边推理。
VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
- 用强化学习训练视觉语言模型，交替生成文字和中间图像推理步骤。
- 在图表问答任务中提升性能，准确率显著优于仅依赖静态图像的模型。
- 首次实现无过程监督的多模态链式思维，适合研究多轮跨模态推理的人。
强化学习微调（RFT）通过长链思维、自我修正和有效工具使用，显著提升了大语言模型的推理能力。尽管近期工作尝试将RFT扩展到视觉语言模型（VLMs），但这些方法大多产生仅依赖静态图像输入的纯文本推理，未能实现真正的多模态推理。相比之下，测试时的方法如Visual Sketchpad虽包含视觉步骤，却缺乏训练机制。我们提出VTool-R1，首个通过在多模态工具使用中融合强化学习，训练VLM生成文本与中间视觉推理步骤交错的多模态链式思维的框架。VTool-R1将基于Python的视觉编辑工具整合进RFT流程，使VLM学会在何时及如何生成有助于最终推理的视觉步骤。采用基于任务准确率的成果奖励机制，该方法在不依赖过程监督的情况下，激发策略性视觉工具使用。在图表与表格上的结构化视觉问答实验表明，VTool-R1通过教会VLM‘用图像思考’，显著提升了推理表现。为支持未来多轮多模态推理研究，我们开源代码于https://github.com/VTOOL-R1/vtool-r1。
原文摘要 · Abstract (English)
Reinforcement Learning Finetuning (RFT) has significantly advanced the reasoning capabilities of large language models (LLMs) by enabling long chains of thought, self-correction, and effective tool use. While recent works attempt to extend RFT to vision-language models (VLMs), these efforts largely produce text-only reasoning conditioned on static image inputs, falling short of true multimodal reasoning in the response. In contrast, test-time methods like Visual Sketchpad incorporate visual steps but lack training mechanisms. We introduce VTool-R1, the first framework that trains VLMs to generate multimodal chains of thought by interleaving text and intermediate visual reasoning steps. VTool-R1 integrates Python-based visual editing tools into the RFT process, enabling VLMs to learn when and how to generate visual reasoning steps that benefit final reasoning. Trained with outcome-based rewards tied to task accuracy, our approach elicits strategic visual tool use for reasoning without relying on process-based supervision. Experiments on structured visual question answering over charts and tables show that VTool-R1 enhances reasoning performance by teaching VLMs to "think with images" and generate multimodal chain of thoughts with tools. To support future research in multi-turn multi-modal reasoning, we open-source our code at https://github.com/VTOOL-R1/vtool-r1
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。