让视觉语言模型学会灵活使用多种工具进行精准空间推理
SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL
- 采用双阶段强化学习,让模型通过互动探索掌握多工具协作
- 在多个基准上超越现有方法,机器人实操成功率显著提升
- 适合需要精确空间理解的机器人控制与智能体应用
视觉语言模型(VLMs)具备出色的视觉理解能力,但在具身应用中缺乏精确的空间推理能力。代理范式有望通过引入深度估计、分割模型、位姿估计等工具来增强其能力,但如何在不依赖手工提示或固定工具链的前提下实现最优工具使用仍是一个挑战。以往的强化学习受限于多工具推理的庞大搜索空间,仅能处理单一视觉工具。本文提出双交互强化学习(DIRL),分两阶段训练:教学阶段结合单工具专家的示范与全工具前沿模型的轨迹;探索阶段通过持续强化学习优化多工具协调。所提出的SpaceTools模型在空间理解基准(RoboSpatial-Home、BLINK、BOP-ASK)上达到最先进性能,并在7自由度机器人上实现了可靠的现实操作。相比基础微调(SFT)提升12%,相比纯强化学习(RL)提升16%。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) demonstrate strong qualitative visual understanding, but struggle with metrically precise spatial reasoning required for embodied applications. The agentic paradigm promises that VLMs can use a wide variety of tools that could augment these capabilities, such as depth estimators, segmentation models, and pose estimators. Yet it remains an open challenge how to realize this vision without solely relying on handcrafted prompting strategies or enforcing fixed, predefined tool pipelines that limit VLMs' ability to discover optimal tool-use patterns. Reinforcement Learning could overcome this gap, but has so far been limited to reasoning with a single visual tool due to the large search space in multi-tool reasoning. We introduce Double Interactive Reinforcement Learning (DIRL), a two-phase training framework where VLMs learn to coordinate multiple tools through interactive exploration and feedback. In the teaching phase, we combine demonstrations from a single tool specialist trained via interactive RL with traces from a frontier model using all tools. In the exploration phase, the model further refines multi-tool coordination through continued RL. Our model, SpaceTools, with tool-augmented spatial reasoning ability, achieves state-of-the-art performance on spatial understanding benchmarks (RoboSpatial-Home, BLINK, BOP-ASK) and demonstrates reliable real-world manipulation using a 7-DOF robot as a tool. DIRL provides substantial improvements over the vanilla SFT (+12% on RoboSpatial) and RL (+16% on RoboSpatial) baselines. Project page: https://spacetools.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。