arXiv:2603.12939cs.RO2026-03中稿 · ECCV被引 4

让机器人具备持续记忆与因果推理能力,提升长时操作可靠性

RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics

  • 用时空融合令牌绑定视觉与3D几何信息,实现物体持久定位
  • 构建因果时空图记录动作引发的状态变化,避免感知误差累积
  • 无需训练即可在复杂任务中保持物体恒常性,适合高阶机器人控制

实现可靠的长时程机器人操作是迈向开放世界具身智能的关键一步。然而,基于视觉语言模型(VLM)的规划器将每一步视为独立的观察-动作映射,导致其在每个决策点都需从原始像素重新推断场景几何,且无法感知先前动作对环境的改变。尽管短期表现优异,但缺乏持续的时空推理能力,难以维持几何锚定和动作触发的状态记忆。缺少持续状态追踪会导致感知误差随执行过程累积,被暂时遮挡的物体被彻底遗忘,进而引发预条件违规并逐级放大。相比之下,人类能持续维护心理模型,跟踪空间关系与动作后果。受此启发,我们提出RoboStream——一种无需训练的框架,通过时空融合令牌(STF-Tokens)将视觉证据与3D几何属性绑定,实现物体持久定位;并通过因果时空图(CSTG)记录跨步骤的动作触发状态转换,维持因果连续性。该设计使规划器能够追溯因果链,在遮挡下仍保持物体恒常性,无需额外训练或微调。RoboStream在长时程RLBench任务中达到90.5%成功率,在挑战性的真实世界积木搭建任务中达44.4%,而SoFar和VoxPoser均仅11.1%,证明时空推理与因果记忆是可靠长时操作的关键缺失组件。

原文摘要 · Abstract (English)

Enabling reliable long-horizon robotic manipulation is a crucial step toward open-world embodied intelligence. However, VLM-based planners treat each step as an isolated observation-to-action mapping, forcing them to reinfer scene geometry from raw pixels at every decision point while remaining unaware of how prior actions have reshaped the environment. Despite strong short-horizon performance, these systems lack the spatio-temporal reasoning required for persistent geometric anchoring and memory of action-triggered state transitions. Without persistent state tracking, perceptual errors accumulate across the execution horizon, temporarily occluded objects are catastrophically forgotten, and these compounding failures lead to precondition violations that cascade through subsequent steps. In contrast, humans maintain a persistent mental model that continuously tracks spatial relations and action consequences across interactions rather than reconstructing them at each instant. Inspired by this human capacity for causal spatio-temporal reasoning with persistent memory, we propose RoboStream, a training-free framework that achieves geometric anchoring through Spatio-Temporal Fusion Tokens (STF-Tokens), which bind visual evidence to 3D geometric attributes for persistent object grounding, and maintains causal continuity via a Causal Spatio-Temporal Graph (CSTG) that records action-triggered state transitions across steps. This design enables the planner to trace causal chains and preserve object permanence under occlusion without additional training or fine-tuning. RoboStream achieves 90.5% on long-horizon RLBench and 44.4% on challenging real-world block-building tasks, where both SoFar and VoxPoser score 11.1%, demonstrating that spatio-temporal reasoning and causal memory are critical missing components for reliable long-horizon manipulation.

机器人操作时空推理记忆机制视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。