用动作为中心的视觉-动作模型,让机器人决策快9倍且成功率更高。
GigaWorld-Policy: An Efficient Action-Centered World--Action Model
- 以动作为中心建模,分离视觉与动作预测,减少计算负担。
- 推理速度提升9倍,任务成功率提高7%,在真实机器人上表现更优。
- 可选生成视频,部署时仅需快速预测动作,适合实时控制场景。
基于预训练视频生成模型的世界-动作模型(WAM)在机器人策略学习中展现出巨大潜力。然而,现有方法存在两大瓶颈:一是同时推理未来视觉动态与对应动作导致推理开销大;二是视觉与运动表示纠缠,使动作预测依赖于未来视频生成质量。为此,我们提出GigaWorld-Policy,一种以动作为中心的世界-动作模型,能够学习2D像素级动作动态,并支持高效动作解码,可选视频生成。该模型将策略训练分为两个耦合组件:在当前观测下预测未来动作序列,同时根据预测动作和相同观测生成未来视频。策略受动作预测和视频生成双重监督,通过视觉-动态约束增强动作物理合理性。采用因果设计,防止未来视频令牌影响动作令牌,使未来视频生成在推理时可选,实现部署时更快的动作预测。为支持该范式,我们构建了一个多样、大规模的机器人数据集用于预训练动作中心视频生成模型,并将其作为机器人策略学习的骨干。实验证明,GigaWorld-Policy在真实机器人平台上运行速度比领先基线Motus快9倍,任务成功率提升7%;相较于pi-0.5,其在RoboTwin 2.0上的性能提升达95%。
原文摘要 · Abstract (English)
World-Action Models (WAM) initialized from pre-trained video generation backbones have demonstrated remarkable potential for robot policy learning. However, existing approaches face two critical bottlenecks that hinder performance and deployment. First, jointly reasoning over future visual dynamics and corresponding actions incurs substantial inference overhead. Second, joint modeling often entangles visual and motion representations, making motion prediction accuracy heavily dependent on the quality of future video forecasts. To address these issues, we introduce GigaWorld-Policy, an action-centered WAM that learns 2D pixel-action dynamics while enabling efficient action decoding, with optional video generation. Specifically, we formulate policy training into two coupled components: the model predicts future action sequences conditioned on the current observation, and simultaneously generates future videos conditioned on the predicted actions and the same observation. The policy is supervised by both action prediction and video generation, providing richer learning signals and encouraging physically plausible actions through visual-dynamics constraints. With a causal design that prevents future-video tokens from influencing action tokens, explicit future-video generation is optional at inference time, allowing faster action prediction during deployment. To support this paradigm, we curate a diverse, large-scale robot dataset to pre-train an action-centered video generation model, which is then adapted as the backbone for robot policy learning. Experimental results on real-world robotic platforms show that GigaWorld-Policy runs 9x faster than the leading WAM baseline, Motus, while improving task success rates by 7%. Moreover, compared with pi-0.5, GigaWorld-Policy improves performance by 95% on RoboTwin 2.0.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。