用视频模型生成动作,实现跨机器人零样本控制。
Turning Video Models into Generalist Robot Policies

- 视频模型不改,只加特定机器人的逆动力学模型来转动作。
- 在仿真和真实机器人上表现优异,支持16自由度灵巧手操作。
- 无需重新训练,换机器人只需换逆模型,适合快速部署。
视频生成模型已成为机器人领域的有力基础,能生成描述复杂任务完成过程的视频。现有方法通过微调带动作标注的数据联合预测未来观测与动作。本文提出新思路:保持视频规划器不变,仅训练针对具体机器人构型的逆动力学模型(IDM)。该解耦设计使视频规划器保持泛化性,可灵活替换不同视频模型且无需重训IDM,IDM亦可利用自对弈数据独立训练。我们构建了闭环的视频到动作策略,结合无动作视频世界模型与基于机器人雅可比矩阵设计的IDM。结果表明该方法在高维动作空间中数据效率高、可扩展性强。所提出的视频到具身机器人动作模型(VERA)在模拟与真实世界基准测试中表现良好,包括零样本的Panda机械臂操作和16自由度Allegro手的立方体翻转任务。同一视频规划器可通过搭配不同专属IDM用于多种机器人。实验表明,解耦视频规划与精确动作转换是实现零样本、跨机器人、通用机器人控制的有效路径。
原文摘要 · Abstract (English)
Video generative models have emerged as a promising robotics backbone, capable of generating videos that depict the completion of complex tasks across embodiments and environments. Recent work proposes robot foundation models that jointly predict future observations and actions by finetuning video models with action-labeled data. In this paper, we test the limits of an alternative approach: leave the video planner as-is while training an embodiment-specific inverse dynamics model (IDM). This decoupling offers several natural benefits: the video planner remains embodiment-agnostic, different video models can be interchanged easily without re-training the IDM, and the IDM can be independently trained with readily available self-play data. We present a closed-loop, video-to-action policy that combines an action-free video world model with a carefully-designed IDM based on the robot embodiment Jacobian. We demonstrate that our IDM design is both data-efficient and scalable to high-dimensional action spaces. Our policy, which we coin the Video-to-Embodied Robot Action Model (VERA), achieves strong performance across simulated and real-world benchmarks, including zero-shot Panda arm manipulation and 16-DoF Allegro-hand dexterous cube re-orientation. The same video planner can be used across multiple embodiments by pairing it with different embodiment-specific IDMs. Our results show that decoupled video planning plus faithful video-to-action translation is a viable alternative route towards zero-shot, cross-embodiment, and generalizable robot control. More results are available on our project website: https://vera.csail.mit.edu.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。