arXiv:2602.23721cs.ROcs.CV2026-02被引 1

让机器人预测动作时提前构想3D空间和历史动态,提升复杂任务表现。

StemVLA:An Open-Source Vision-Language-Action Model with Future 3D Spatial Geometry Knowledge and 4D Historical Representation

  • 用未来3D几何结构和4D历史时空表示增强视觉语言动作模型
  • 在LIBERO数据集上达到92.0%准确率,长程任务达86.0%
  • 适合需要空间推理与长期规划的机器人操控研究者

视觉-语言-动作(VLA)模型通过结合视觉观测与语言指令来预测机器人动作,在操作任务中展现出良好泛化能力。然而,现有方法大多依赖从2D视觉输入到动作序列的直接映射,未显式建模潜在的3D空间结构或时间世界动态,限制了在动态环境中的空间推理与长时决策能力。为此,我们提出StemVLA,一种新框架,显式融合未来导向的3D空间几何知识与历史4D时空表示。首先,不依赖仅有的图像观测,StemVLA预测结构化的未来3D空间几何世界知识,使模型能预判即将出现的场景结构与物体配置。其次,为捕捉时间一致性与运动动态,将历史图像帧输入预训练的视频-几何变换器主干网络,提取隐式3D世界表示,并通过时间注意力模块(VideoFormer)跨时间聚合,形成统一的4D历史时空表示。通过联合建模2D观测、预测的3D未来结构与聚合的4D时间动态,StemVLA实现对机器人操作更全面的世界理解。大量仿真实验表明,StemVLA在LIBERO各子集上平均准确率达92.0%,在长时程任务LIBERO-Long上达86.0%。

原文摘要 · Abstract (English)

Vision-language-action (VLA) models integrate visual observations and language instructions to predict robot actions, demonstrating promising generalization in manipulation tasks. However, most existing approaches primarily rely on direct mappings from 2D visual inputs to action sequences, without explicitly modeling the underlying 3D spatial structure or temporal world dynamics. Such representations may limit spatial reasoning and long-horizon decision-making in dynamic environments. To address this limitation, we propose StemVLA, a novel framework that explicitly incorporates both future-oriented 3D spatial knowledge and historical 4D spatiotemporal representations into action prediction. First, instead of relying solely on observed images, StemVLA forecasts structured 3D future spatial-geometric world knowledge, enabling the model to anticipate upcoming scene geometry and object configurations. Second, to capture temporal consistency and motion dynamics, we feed historical image frames into a pretrained video-geometry transformer backbone to extract implicit 3D world representations, and further aggregate them across time using a temporal attention module, termed VideoFormer [20], forming a unified 4D historical spatiotemporal representation. By jointly modeling 2D observations, predicted 3D future structure, and aggregated 4D temporal dynamics, StemVLA enables more comprehensive world understanding for robot manipulation. Extensive experiments in simulation demonstrate that Stem-VLA achieves an average accuracy of 92.0% across the LIBERO subsets, and 86.0% on the long-horizon LIBERO-Long subset.

机器人操控多模态模型3D预测时空建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。