arXiv:2606.13673cs.CVcs.AI2026-06被引 5

用可执行代码作为动作接口,让智能体灵活处理3D空间推理任务。

SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning

论文配图:SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning
图 1 · 摘自论文原文
  • 以可执行的Python代码为动作接口,支持逐步更新和动态调整推理过程。
  • 在20个基准上平均准确率达59.9%,比最新方法提升11.2个百分点。
  • 无需微调,适配多种视觉语言模型,适合复杂3D/4D空间推理场景。

空间推理——即判断物体位置、相互关系及三维运动的能力——仍是视觉-语言模型(VLMs)的核心挑战。现有工具增强型智能体通过引入专用感知模块来应对,但其性能受限于工具调用的动作接口设计。当前方法要么采用单次代码执行,需提前确定完整分析策略;要么依赖结构化工具调用接口,灵活性不足。两者均难以支撑开放式的复杂3D/4D空间推理。为此,我们提出SpatialClaw:一种无需训练的空间推理框架,采用代码作为动作接口。SpatialClaw维护一个预加载输入帧与感知、几何原语的有状态Python内核,使基于VLM的智能体可在每一步根据所有先前输出编写可执行代码单元,从而灵活组合和操作感知结果,并根据中间文本、视觉观察及任务需求动态调整分析策略。在涵盖静态与动态3D/4D空间推理任务的20个基准上评估,SpatialClaw平均准确率达59.9%,相较近期方法提升11.2个百分点,且在六种不同VLM骨干网络上均表现一致,无需针对具体任务或模型进行适配。

原文摘要 · Abstract (English)

Spatial reasoning, the ability to determine where objects are, how they relate, and how they move in 3D, remains a fundamental challenge for vision-language models (VLMs). Tool-augmented agents attempt to address this by augmenting VLMs with specialist perception modules, yet their effectiveness is bounded by the action interface through which those tools are invoked. In this work, we study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning. Existing spatial agents either employ single-pass code execution, which commits to a full analysis strategy before any intermediate result is observed, or rely on a structured tool-call interface that often offers less flexibility for freely composing operations or tailoring the analysis to each task. Both designs offer limited flexibility for open-ended, complex 3D/4D spatial reasoning. We therefore propose SpatialClaw, a training-free framework for spatial reasoning that adopts code as the action interface. SpatialClaw maintains a stateful Python kernel pre-loaded with input frames and a suite of perception and geometry primitives, letting a VLM-backed agent write one executable cell per step conditioned on all prior outputs, enabling the agent to flexibly compose and manipulate perception results and adapt its analysis to both intermediate text and visual observations and the demands of each problem. Evaluated across 20 spatial reasoning benchmarks spanning a broad range of static and dynamic 3D/4D spatial reasoning tasks, SpatialClaw achieves 59.9% average accuracy, outperforming the recent spatial agent by +11.2 points, with consistent gains across six VLM backbones from two model families without any benchmark- or model-specific adaptation.

空间推理智能体代码接口多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。