让机器人模型在不生成未来画面的情况下预测动作,提升效率与准确性。
Foresight Without Seeing: Latent Futures for World Action Models

- 用隐状态捕捉未来动态,直接指导动作生成,无需反复解码视频。
- 在LIBERO数据集上达到96.7%成功率,LIBERO-Plus达61.6%。
- 适合需要高效决策的机器人任务,尤其无真实交互数据时仍表现优异。
世界动作模型(WAMs)将未来视觉预测与机器人动作生成结合,使策略能够建模交互过程中物理世界的演变。现有WAMs在预测动态如何暴露给动作路径方面存在差异:显式未来模型可直接访问预测场景演化,但需多次视频去噪带来高推理成本;直接策略模型能高效从当前观测预测动作,却缺乏推理时暴露预测动态的接口。为此,我们提出ForeWAM,一种基于动态条件的直接策略WAM,可在不解码未来视频的前提下为动作生成提供预测上下文。核心是Future-KV对当前视觉隐状态和随机未来槽位进行一次视频DiT预填充,复用生成的层间键值状态完成动作去噪。此外引入由冻结隐动作教师监督的动力学寄存器,促使隐式未来状态捕捉物体运动、接触变化和任务进展等交互引起的转变。训练阶段使用真实未来观测和教师模型;部署时无需二者,也不生成未来视频。未使用实体机器人数据预训练的情况下,标准与加速版本的ForeWAM在LIBERO上分别取得96.7%和96.9%的平均成功率,标准版本在LIBERO-Plus上达61.6%。结果表明,直接策略WAM可在保持高效动作预测的同时,无需显式生成未来观测即可向动作路径暴露预测动态。
原文摘要 · Abstract (English)
World Action Models (WAMs) couple future visual prediction with robot action generation, enabling policies to model how the physical world evolves during interaction. Existing WAMs differ in how predictive dynamics are exposed to the action pathway. Explicit-future WAMs provide direct access to predicted scene evolution, but incur substantial inference costs from iterative video denoising. In contrast, direct-policy WAMs efficiently predict actions from the current observation but lack an explicit inference-time interface for exposing predictive dynamics to the Action DiT. To bridge this gap, we propose ForeWAM, a dynamics-conditioned direct-policy WAM that provides predictive context for action generation without decoding future videos. At its core, Future-KV performs a single Video DiT prefill over the current visual latent and stochastic future slots, and reuses the resulting layer-wise key-value states throughout action denoising. We further introduce dynamics registers supervised by a frozen latent action teacher, encouraging the implicit future states to capture interaction-induced transitions such as object motion, contact changes, and task progress. Ground-truth future observations and the teacher are used only during training; deployment requires neither and performs no future video generation. Without embodied robot data pretraining, the standard and accelerated variants of ForeWAM achieve average success rates of 96.7% and 96.9% on LIBERO, respectively. The standard variant further achieves 61.6% success on LIBERO-Plus. These results demonstrate that direct-policy WAMs can retain efficient action prediction while exposing predictive dynamics to the action pathway without explicitly generating future observations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。