让视觉语言模型学会像人一样用工具思考图像。
OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning
- 设计标准化工具接口与强化学习框架,让模型自主选择最佳图像工具
- 在图表推理任务中比监督微调提升28.83分,超越GPT-4.1 8.68分
- 开源完整流程,适合研究动态视觉推理与智能体构建的开发者
人类能灵活运用交互式视觉认知解决复杂问题,但让大视觉语言模型(LVLMs)学习类似的自适应行为仍面临挑战。主要瓶颈在于缺乏标准化基础设施,难以整合多样工具、生成丰富交互数据并有效训练智能体。为此,我们提出OpenThinkIMG,首个开源的端到端工具增强型LVLM框架,包含标准化视觉工具接口、可扩展轨迹生成与灵活训练环境。针对监督微调在动态工具调用上泛化能力有限的问题,我们提出新型强化学习框架V-ToolRL,使LVLM通过工具交互反馈直接优化任务成功率,自主发现最优工具使用策略。我们在挑战性图表推理任务上验证了V-ToolRL,基于Qwen2-VL-2B的强化学习智能体显著优于其监督微调初始化版本(+28.83分),平均超越Taco和CogCom等监督基线+12.7分,且超过闭源模型GPT-4.1达+8.68分。我们希望OpenThinkIMG能成为推动动态、工具增强型视觉推理的基础框架,助力社区构建真正能‘以图思考’的AI智能体。
原文摘要 · Abstract (English)
While humans can flexibly leverage interactive visual cognition for complex problem-solving, enabling Large Vision-Language Models (LVLMs) to learn similarly adaptive behaviors with visual tools remains challenging. A significant hurdle is the current lack of standardized infrastructure, which hinders integrating diverse tools, generating rich interaction data, and training robust agents effectively. To address these gaps, we introduce OpenThinkIMG, the first open-source, comprehensive end-to-end framework for tool-augmented LVLMs. It features standardized vision tool interfaces, scalable trajectory generation for policy initialization, and a flexible training environment. Furthermore, considering supervised fine-tuning (SFT) on static demonstrations offers limited policy generalization for dynamic tool invocation, we propose a novel reinforcement learning (RL) framework V-ToolRL to train LVLMs to learn adaptive policies for invoking external vision tools. V-ToolRL enables LVLMs to autonomously discover optimal tool-usage strategies by directly optimizing for task success using feedback from tool interactions. We empirically validate V-ToolRL on challenging chart reasoning tasks. Our RL-trained agent, built upon a Qwen2-VL-2B, significantly outperforms its SFT-initialized counterpart (+28.83 points) and surpasses established supervised tool-learning baselines like Taco and CogCom by an average of +12.7 points. Notably, it also surpasses prominent closed-source models like GPT-4.1 by +8.68 accuracy points. We hope OpenThinkIMG can serve as a foundational framework for advancing dynamic, tool-augmented visual reasoning, helping the community develop AI agents that can genuinely "think with images".
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。