arXiv:2606.20515cs.CV2026-06被引 3

S-Agent让模型像人一样在3D空间中用工具推理,突破静态图像局限。

S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence

论文配图:S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence
图 1 · 摘自论文原文
  • 用时空证据累积替代单帧判断,构建场景级空间理解。
  • 通过2D到3D的工具链实现物体定位与测量,准确率提升显著。
  • 适合需要空间推理的视觉智能任务,如机器人导航与环境建模。

真实世界的空间智能需要对连续演变的3D世界进行推理,但现有视觉语言模型(VLM)和工具增强型代理仍依赖孤立视觉观察的静态、无状态推断。我们提出S-Agent,一种面向连续多视角图像与视频的空间推理范式。将空间推理重构为时空证据积累过程,而非单帧预测,使空间感知从帧中心识别转向以场景为中心的理解。S-Agent将VLM作为语义规划器,决定所需证据;通过多层级空间工具与专家,将物体在2D中定位,升维至3D几何证据,并聚合为高层空间知识(如计数、测量、朝向与相对位置)。同时引入时间记忆机制:场景记忆维持演化中的场景状态,代理记忆积累推理上下文,支持跨帧与推理步骤的证据融合。在多视角与视频空间推理基准上的全面实验表明,S-Agent以无需训练的方式持续提升开源与闭源VLM性能。此外,在S-Agent生成的空间轨迹数据集S-300K上进行监督微调,得到的S-Agent-8B模型在小型模型中表现超越同类(如Qwen3-VL-8B),并接近先进闭源模型(如GPT-5.4和Gemini 3)水平。

原文摘要 · Abstract (English)

Real-world spatial intelligence requires reasoning over a continuous and evolving 3D world, yet existing VLMs and tool-augmented agents largely remain tied to static, stateless inference from isolated visual observations. We introduce \textbf{\textsc{S-Agent}}, a spatial tool-use agentic paradigm for understanding and reasoning over continuous multi-view images and videos. By formulating spatial reasoning as spatio-temporal evidence accumulation rather than isolated frame-level prediction, \textsc{S-Agent} reshapes spatial perception into scene-centric understanding beyond frame-centric recognition. Specifically, \textsc{S-Agent} casts the VLM as a semantic planner that decides what evidence is needed, while a hierarchy of spatial tools and experts grounds objects in 2D, lifts them into 3D geometric evidence, and aggregates this evidence into high-level spatial knowledge (\textit{e.g.}, counting, measurement, orientation, and relative position). Additionally, a temporal memory mechanism, including Scene Memory for maintaining the evolving scene state and Agent Memory for accumulating reasoning context, enables evidence integration across frames and reasoning steps. Comprehensive experiments on multi-view and video spatial reasoning benchmarks show that \textsc{S-Agent} consistently improves both open-source and closed-source VLMs in a training-free manner. Beyond inference-time augmentation, supervised fine-tuning (SFT) on \textsc{S-Agent}-generated spatial trajectories \textsc{S-300K} yields \textsc{S-Agent-8B}, a compact spatial agent that significantly surpasses similar-scale baselines (e.g., Qwen3-VL-8B) and performs comparably to advanced closed-source models (e.g., GPT-5.4 and Gemini 3).

空间推理视觉语言模型工具使用多视角理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。