让单帧视觉语言模型具备流式时序推理能力,无需增加参数。
StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models

- 以指令-视觉对为时间单元,用双向与因果注意力实现跨模态融合与流式推理。
- 在真实机器人任务中表现优于pi0.5,支持更快速平滑的动作执行。
- 适合需要记忆和精确空间感知的机器人控制场景,兼容异步部署。
视觉-语言-动作(VLA)模型在机器人操作中表现出色,但现有先进模型如pi0.5采用单帧范式,难以保留历史观测并建立精准的空间感知。本文提出StreamPI,一种流式多模态时序建模框架,使单帧VLA具备时序推理能力且不引入额外参数。核心设计为指令锚定的时序建模:将每个(视觉观测,语言指令)对视为原子时间单元,对内使用双向注意力实现跨模态融合,跨对使用因果注意力保持自回归流式推断,确保语言指令在任务执行中持续作为语义锚点。为弥合同步训练与异步真实机器人部署之间的差距,提出随机间隔流式训练策略:合理帧间间隔(如每3帧)可提升动作执行速度与流畅性;进一步随机化间隔可增强对帧率扰动的鲁棒性,支持实际异步部署。此外,借助大语言模型(LLM)骨干的长度外推能力,StreamPI无缝继承预训练单帧权重,支持灵活的单帧与多帧推理。在涵盖记忆依赖与精确感知场景的真实机器人任务及仿真基准LIBERO上的实验表明,StreamPI在多样任务中均超越pi0.5。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have demonstrated effectiveness in robot manipulation, yet state-of-the-art models such as pi0.5 operate under a single-frame paradigm, limiting their ability to retain past observations and develop precise spatial perception. In this paper, we propose StreamPI, a streaming multimodal temporal modeling framework that equips single-frame VLA with temporal reasoning capability without introducing any additional parameters. One core design is instruction-anchored temporal modeling. It treats each (visual observation, language instruction) pair as an atomic temporal unit: bidirectional attention within each pair enables cross-modal fusion, while causal attention across pairs preserves autoregressive streaming inference. This ensures the language instruction serves as a persistent semantic anchor throughout task execution. To bridge the gap between synchronous training and asynchronous real-robot deployment, we introduce a andom-interval streaming training strategy: a proper inter-frame interval (e.g., every 3 frames) enables faster and smoother action execution. Beyond this, randomizing the interval further improves robustness to frame-timing perturbations, supporting asynchronous deployment in practice. Furthermore, by leveraging the length extrapolation capability of the LLM backbone, StreamPI seamlessly inherits pretrained single-frame weights and supports flexible single-frame and multi-frame inference. Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。