将视频生成模型用于移动机器人操控，实现全身协同运动与操作。
MobileWAM: Bridging World Action Models to Mobile Manipulation with Chain-of-Foresight
- 用分层注意力融合预训练视频模型与轻量动作专家，实现端到端控制。
- 在ManiSkill-HAB上超越现有最先进方法，实机验证展现强泛化能力。
- 适合研究移动机器人、具身智能及多模态动作建模的开发者参考。
基于视频生成主干的世界动作模型（WAM）是机器人学习的新范式，但目前仅限于桌面级操作。移动操纵需在场景尺度动态下同步完成移动与全身操作，而现有方法仍依赖无视动态的视觉编码器和手工协调。本文提出MobileWAM，一种混合变换器架构，通过分层联合注意力融合预训练视频扩散变换器与轻量动作专家，将互联网规模的动作先验转化为全身控制。为应对移动与操作的异构动态，动作专家的每层变为共享、移动与操作三类专家的混合，由动作标记中的运动意图软路由。为增强监督，提出链式预见（Chain-of-Foresight, CoF）：中间表示依次预测一系列未来潜在块，每步依赖前序结果。CoF 与解耦的视频-动作去噪方案天然适配。部署时，WAM仅作当前帧编码器；预见链与视频生成仅在训练中使用，推理时丢弃，仅保留策略级开销。MobileWAM 在 ManiSkill-HAB 上超越现有最优方法，并在真实 ARX Lift2 移动机械臂上跨任务微调，表现出优异泛化能力。代码即将开源。
原文摘要 · Abstract (English)
World action models (WAMs) built on video generation backbones are a rising recipe for robot learning, yet remain confined to tabletop manipulation. Mobile manipulation demands simultaneous locomotion and whole-body manipulation amid scene-scale dynamics, yet is still dominated by dynamics-blind visual encoders with hand-crafted coordination. We bridge this gap with MobileWAM, a mixture-of-transformers architecture that fuses a pretrained video diffusion transformer with a lightweight action expert through layerwise joint attention, translating internet-scale motion priors into whole-body control. To reconcile the heterogeneous dynamics of moving and manipulating, each feed-forward layer of the action expert becomes a three-expert mixture of shared, locomotion, and manipulation experts, softly routed by the motion intent in the action tokens. To densify supervision, we further propose Chain-of-Foresight (CoF): intermediate representations sequentially predict a chain of future latent chunks, each step conditioned on its predecessor. CoF pairs naturally with our decoupled video--action denoising scheme. At deployment, the WAM serves as a pure current-frame encoder; foresight acts only through gradients, so at inference the foresight chain and video generation are discarded, leaving only policy-level cost. MobileWAM surpasses state-of-the-art mobile manipulation policies on ManiSkill-HAB and fine-tunes to a real ARX Lift2 mobile manipulator across diverse tasks with strong generalization. Code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。