让大模型用工具提示来提升空间推理能力,减少错误。
LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models
- 用统一接口调用多种视觉工具,生成可直接使用的多模态提示。
- 在4个数据集上比基础模型高20%性能,超越部分闭源大模型。
- 适合需要精准空间理解的场景,如机器人导航、智能设计。
空间推理是智能系统感知和交互物理世界的核心能力。然而,多模态大语言模型(MLLMs)在解析复杂几何布局时经常出现幻觉和不精确问题。由于数据驱动的扩展难以内化结构化的几何先验和空间约束,引入成熟的专用视觉模型成为可行替代方案。但该范式在空间推理中面临两大挑战:难以调用异构且参数丰富的工具,以及难以理解并有效利用其多样化的低层输出(如分割掩码、深度图)进行高层推理。为此,我们提出LAST,一种工具增强型空间推理的统一框架。LAST包含一个可扩展的交互沙箱(LAST-Box),将异构工具调用抽象为原子指令和可复用的空间技能,并返回可被大模型直接消费的多模态提示(如标注图像和文本描述)。我们进一步设计三阶段渐进式训练策略,引导模型从理解工具输出到熟练且自适应地调用工具。在四个数据集上的实验表明,LAST-7B相比其基线模型提升约20%,并优于强健的闭源大模型,在复杂空间任务上显著增强推理能力。
原文摘要 · Abstract (English)
Spatial reasoning is a cornerstone capability for intelligent systems to perceive and interact with the physical world. However, multimodal large language models (MLLMs) frequently suffer from hallucinations and imprecision when parsing complex geometric layouts. As data-driven scaling struggles to internalize structured geometric priors and spatial constraints, integrating mature, specialized vision models presents a compelling alternative. Despite its promise, applying this paradigm to spatial reasoning is hindered by two key challenges: The difficulty of invoking heterogeneous, parameter-rich tools, as well as the challenge of understanding and effectively leveraging their diverse low-level outputs (e.g., segmentation masks, depth maps) in high-level reasoning. To address these challenges, we propose LAST, a unified framework for tool-augmented spatial reasoning. LAST features an extensible interactive sandbox, termed LAST-Box, which abstracts heterogeneous tool invocations into atomic instructions and reusable spatial skills, returning multimodal hints (e.g., annotated images and textual descriptions) that can be directly consumed by LLMs. We further design a three-stage progressive training strategy that guides models from understanding tool outputs to proficient and adaptive tool invocation. Experiments on four datasets show that LAST-7B achieves around 20\% performance gains over its backbone and outperforms strong proprietary closed-source LLMs, substantially enhancing reasoning on complex spatial tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。