DreamWAM通过预测非RGB的多模态未来状态,提升机器人动作模型的鲁棒性。
DreamWAM: Beyond RGB Future Prediction for World Action Models

- 将未来预测从纯RGB扩展到外观、运动、几何与语义的联合建模
- 在LIBERO上提升成功率至98.90%,在扰动下达75.47%
- 适合需要应对光照、背景变化的真实世界机器人任务
世界动作模型(WAMs)通过预测世界演化来学习与动作相关的表征。现有方法多依赖RGB空间中的未来预测,导致任务相关状态变化与纹理、光照、背景和视角等干扰因素纠缠。本文提出DreamWAM,将未来预测重构为超越RGB的结构化世界建模,通过外观、运动、几何和语义的互补视图表示未来状态。训练时,结合RGB与运动的联合潜在去噪,辅以轻量级门控残差分支处理几何与语义;视频DiT与动作DiT共享注意力机制,使动作分支可学习未来状态预测。推理时仅输出RGB,其余分支关闭。在无滚动和联合视频-动作推理中,DreamWAM在LIBERO上分别达到98.40%和98.90%,优于基线。在未见的LIBERO-Plus扰动下,成功率达63.44%和75.47%。真实世界操作中,面对光照、背景与物体布局变化,平均成功率74.4%,显著高于Fast-WAM-Joint的55.6%。结果表明,鲁棒的世界-动作学习不仅依赖未来预测,更取决于以对动作有意义的方式表征未来。代码与模型已开源。
原文摘要 · Abstract (English)
World Action Models (WAMs) learn action-relevant representations by predicting how the observed world will evolve. Most existing WAMs define this future in RGB space, where task-relevant state transitions are entangled with nuisance variations in texture, illumination, background, and viewpoint. We argue that WAMs should explicitly predict action-relevant future state rather than relying on RGB prediction alone. We introduce DreamWAM, which reformulates future prediction as structured world modeling beyond RGB, representing future states through complementary views of appearance, motion, geometry, and semantics. During training, DreamWAM combines joint latent denoising of RGB and motion with lightweight gated residual branches for geometry and semantics. Shared attention between VideoDiT and ActionDiT allows the action branch to learn from these future-state predictions, while all beyond-RGB supervision branches are disabled at inference and deployment remains RGB-only. Across both no-rollout and joint video-action inference, DreamWAM consistently improves the matched RGB-only baselines on LIBERO, from 97.30\% to 98.40\% and from 98.00\% to 98.90\%, respectively. The gains become larger under unseen LIBERO-Plus perturbations, from 51.36\% to 63.44\% and from 69.16\% to 75.47\%. The same robustness extends to real-world manipulation, where DreamWAM attains an average success rate of 74.4\% across unseen changes in lighting, background, and object layout, compared with 55.6\% for Fast-WAM-Joint. These results show that robust world-action learning depends not only on predicting the future, but on representing it in a form that matters for action. The code and models are publicly released at https://github.com/hustvl/DreamWAM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。