arXiv:2603.23864cs.CV2026-03被引 2

提出新基准与模型,让智能体像人一样边看边探索,实时回答空间问题。

See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning

  • 用记忆折叠+主动探索,实现长时间流式推理
  • 在模拟和真实数据上分别提升8.8%和13.3%准确率
  • 适合做机器人、自动驾驶等需要实时感知的场景

空间理解对具身智能体至关重要,但现有视觉语言模型(VLM)和评测基准多为离线评估,忽略两个关键部署需求:长时序流式推理和当前视角不足时的主动感知。为此,我们提出S3-Bench,一个支持主动探索的流式空间问答基准,查询时间戳化,仅使用截至该时刻的观测进行回答。该基准采用双域设计,包含可控制轨迹的可扩展仿真器和包含真实传感噪声的实拍视频流,覆盖10,000+场景与26,000+轨迹,划分专用训练集(S3-Train)和评估集(S3-Eval)。我们进一步提出AMF-VLM,通过(i)记忆折叠——将长序列观测压缩为紧凑结构化记忆;(ii)主动探索——输出移动/旋转/扫描等显式动作获取缺失证据,实现计算受限下的流式空间推理。大量实验表明,在相同训练数据下,相比基线模型,本方法在模拟与真实测试集上分别提升8.8%和13.3%,同时保持对标准空间基准的强迁移能力。

原文摘要 · Abstract (English)

Spatial understanding is fundamental for embodied agents, yet most spatial VLMs and benchmarks remain offline-evaluating post-hoc QA over pre-recorded inputs and overlooking two crucial deployment-critical requirements: long-horizon streaming inference and active perception when the current view is insufficient. To address this gap, we introduce S3-Bench, a benchmark suite for streaming spatial question answering with active exploration, where queries are temporally grounded to specific timestamps and must be answered using only observations available up to that moment. S3-Bench adopts a dual-domain design, combining a scalable simulator with controllable trajectories and exploration actions, and real-world streaming videos that capture practical sensing artifacts for rigorous generalization evaluation. Overall, it spans 10K+ scenes and 26K+ trajectories, with dedicated training (S3-Train) and evaluation (S3-Eval) splits. We further propose AMF-VLM, which supports streaming spatial reasoning under bounded computing via (i) memory folding, which compresses long-horizon observations into compact structured memory, and (ii) active exploration, which outputs explicit actions (e.g. move/rotate/scan) to acquire missing evidence before answering. Extensive experiments demonstrate that, compared to models using identical training data, our approach yields improvements of 8.8% and 13.3% on the simulated and real splits of S3-Eval, respectively, while maintaining competitive transferability to standard spatial benchmarks.

空间理解主动感知流式推理具身智能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。