用视频因果建模让机器人学会预测未来并自主决策。
Causal World Modeling for Robot Control
- 联合视觉与动作的共享潜空间，通过混合变换器架构实现端到端学习。
- 闭环回放机制结合真实观测，提升长期任务控制的稳定性。
- 异步推理支持高效执行，适合复杂场景下的实时机器人控制。
本工作指出，视频世界建模结合视觉语言预训练，为机器人学习建立了全新的独立基础。直观上，视频世界模型可通过理解动作与视觉动态之间的因果关系，实现对近未来状态的想象。受此启发，我们提出LingBot-VA，一种自回归扩散框架，可同时学习帧预测与策略执行。模型包含三项关键设计：(1) 共享潜空间，整合视觉与动作标记，由混合变换器（MoT）架构驱动；(2) 闭环回放机制，持续获取环境反馈并使用真实观测；(3) 异步推理流水线，实现动作预测与电机执行的并行化，支持高效控制。我们在仿真基准和真实场景中评估模型，结果表明其在长时序操作、微调阶段的数据效率以及对新配置的强泛化能力方面表现优异。代码与模型已公开，以促进社区发展。
原文摘要 · Abstract (English)
This work highlights that video world modeling, alongside vision-language pre-training, establishes a fresh and independent foundation for robot learning. Intuitively, video world models provide the ability to imagine the near future by understanding the causality between actions and visual dynamics. Inspired by this, we introduce LingBot-VA, an autoregressive diffusion framework that learns frame prediction and policy execution simultaneously. Our model features three carefully crafted designs: (1) a shared latent space, integrating vision and action tokens, driven by a Mixture-of-Transformers (MoT) architecture, (2) a closed-loop rollout mechanism, allowing for ongoing acquisition of environmental feedback with ground-truth observations, (3) an asynchronous inference pipeline, parallelizing action prediction and motor execution to support efficient control. We evaluate our model on both simulation benchmarks and real-world scenarios, where it shows significant promise in long-horizon manipulation, data efficiency in post-training, and strong generalizability to novel configurations. The code and model are made publicly available to facilitate the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。