用视觉动作建模让机器人在新环境零样本泛化,提升操控鲁棒性。
GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actions

- 基于预训练视频模型,将动作转为视觉表示实现空间可控
- 仅用有限场景数据训练,即在随机新环境中零样本泛化
- 适合需要强泛化能力的机器人政策评估与生成任务
通用机器人策略虽具强大能力,但在复杂未知环境中的鲁棒性仍受限。在多样真实环境中扩展学习与评估成本高且困难。动作条件世界模型虽有前景,但常面临动作控制力弱和分布外(OOD)泛化差的问题。为此,我们提出GeniWorld——一种可泛化的交互式世界模型,用于机器人操作。基于预训练视频生成模型,利用URDF渲染将数值动作转换为视觉动作表示,实现空间接地的动作控制。通过显式解耦机体运动学与环境动态,缓解场景过拟合,促进机器人-环境交互建模。为实现闭环控制,构建了结合高频机器人运动学控制的自回归视频预测模型,支持与机器人策略及人工遥操作交互。实验表明,即使仅在有限固定场景数据上训练,该模型仍取得优异域内性能,并在高度随机的未见环境中实现稳健的零样本泛化。下游应用中,GeniWorld作为可扩展的策略评估器,在环境扰动下仍保持可靠性。此外,即便仅有少量真实示范,其也能在世界模型中生成多样化操作轨迹,提升下游策略在复杂环境中的性能与鲁棒性。
原文摘要 · Abstract (English)
Generalist robot policies exhibit strong capabilities, but their robustness in complex and unseen environments remains limited. Scaling robot learning and evaluation in diverse real-world environments remains costly and challenging. Action-conditioned world models offer a promising alternative, but they often suffer from limited action controllability and poor generalization to out-of-distribution (OOD) scenarios. To this end, we present GeniWorld, an interactive world model for robots that generalizes robustly across unseen scenarios. Building on pretrained video generative models, we use URDF-based rendering to transform numerical actions into visual action representations, enabling spatially grounded action control. By explicitly decoupling embodiment kinematics from environmental dynamics, our model mitigates scene overfitting and facilitates modeling of robot-environment interactions. To achieve closed-loop control, we construct an autoregressive video prediction model integrated with high-frequency robot kinematic control, enabling interaction with both robot policies and human teleoperators. In our experiments, even when trained solely on limited fixed-scene data, our model achieves superior in-domain performance and robust zero-shot generalization to highly randomized, unseen environments. For downstream applications, GeniWorld serves as a scalable policy evaluator that remains reliable under environmental perturbations. Furthermore, even with limited real-world demonstrations, GeniWorld generates diverse manipulation trajectories within the world model, improving downstream policy performance and robustness in complex environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。