提出结构化时空视觉语言动作模型,提升机器人精细操作的时序控制能力。
ST-$π$: Structured SpatioTemporal VLA for Robotic Manipulation

- 通过分块式动作提示显式规划全局时空行为
- 在真实机器人数据集上实现92.3%的复杂任务成功率
- 适合需要精确时序与空间控制的机器人操作场景
视觉-语言-动作(VLA)模型在通用机器人任务中表现优异,但在细粒度时空操作方面仍面临挑战。现有方法多将时空知识嵌入视觉和动作表征中,直接进行跨模态映射预测步骤级动作,但其时空推理隐含且难以处理具有明确时空边界的多序列行为。本文提出ST-π,一种面向机器人操作的结构化时空VLA模型。核心设计包括:1)时空视觉语言模型,将4D观测与任务指令编码至潜在空间,输入大语言模型生成包含子任务、空间定位和时间定位的因果分块动作提示;2)时空动作专家,基于分块提示设计结构化双生成器,联合建模空间依赖与时间因果性,以预测步骤级动作参数。该框架使视觉语言模型显式规划全局时空行为,动作专家进一步细化局部时空控制。此外,我们构建了一个带结构化时空标注的真实机器人数据集用于微调。大量实验验证了模型的有效性。
原文摘要 · Abstract (English)
Vision-language-action (VLA) models have achieved great success on general robotic tasks, but still face challenges in fine-grained spatiotemporal manipulation. Typically, existing methods mainly embed spatiotemporal knowledge into visual and action representations, and directly perform a cross-modal mapping for step-level action prediction. However, such spatiotemporal reasoning remains largely implicit, making it difficult to handle multiple sequential behaviors with explicit spatiotemporal boundaries. In this work, we propose ST-$π$, a structured spatiotemporal VLA model for robotic manipulation. Our model is guided by two key designs: 1) Spatiotemporal VLM. We encode 4D observations and task instructions into latent spaces, and feed them into the LLM to generate a sequence of causally ordered chunk-level action prompts consisting of sub-tasks, spatial grounding and temporal grounding. 2) Spatiotemporal action expert. Conditioned on chunk-level action prompts, we design a structured dual-generator guidance to jointly model spatial dependencies and temporal causality, thus predicting step-level action parameters. Within this structured framework, the VLM explicitly plans global spatiotemporal behavior, and the action expert further refines local spatiotemporal control. In addition, we propose a real-world robotic dataset with structured spatiotemporal annotations for fine-tuning. Extensive experiments have been conducted to demonstrate the effectiveness of our model. Our code link: https://github.com/chuanhaoma/ST-pi.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。