用单阶段微调让视频模型直接生成机器人动作，实现高效智能控制。
Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
- 直接在预训练视频模型潜空间中生成动作和未来状态，无需修改架构。
- 在仿真与真实场景中达成98.5%和67.1%成功率，优于现有方法。
- 支持在线学习与基于模型的规划，适合复杂操控任务研究者。
近期视频生成模型展现出捕捉复杂物理交互与场景时序演化的强大能力。为利用其时空先验，机器人领域尝试将视频模型用于策略学习，但通常需多阶段后训练及新增架构组件来生成动作。本文提出Cosmos Policy，仅通过在目标平台采集的机器人演示数据进行单阶段后训练，即可将大型预训练视频模型（Cosmos-Predict2）转化为有效机器人策略，无需任何架构修改。该方法直接在视频模型的潜扩散过程中生成编码为潜帧的机器人动作，充分利用模型预训练先验与核心学习算法，捕捉复杂动作分布。同时，该策略还生成未来状态图像与价值（预期累计奖励），同样以潜帧形式编码，支持测试时对高成功率动作轨迹的规划。实验表明，Cosmos Policy在LIBERO与RoboCasa仿真基准上分别取得98.5%和67.1%的平均成功率，实现在挑战性双臂操作任务中的最高平均得分，显著优于从头训练的扩散策略、基于视频模型的策略及在相同演示数据上微调的顶尖视觉-语言-动作模型。此外，利用策略回放数据，该方法可自主优化世界模型与价值函数，并借助基于模型的规划进一步提升复杂任务成功率。代码、模型与训练数据已公开于https://research.nvidia.com/labs/dir/cosmos-policy/。
原文摘要 · Abstract (English)
Recent video generation models demonstrate remarkable ability to capture complex physical interactions and scene evolution over time. To leverage their spatiotemporal priors, robotics works have adapted video models for policy learning but introduce complexity by requiring multiple stages of post-training and new architectural components for action generation. In this work, we introduce Cosmos Policy, a simple approach for adapting a large pretrained video model (Cosmos-Predict2) into an effective robot policy through a single stage of post-training on the robot demonstration data collected on the target platform, with no architectural modifications. Cosmos Policy learns to directly generate robot actions encoded as latent frames within the video model's latent diffusion process, harnessing the model's pretrained priors and core learning algorithm to capture complex action distributions. Additionally, Cosmos Policy generates future state images and values (expected cumulative rewards), which are similarly encoded as latent frames, enabling test-time planning of action trajectories with higher likelihood of success. In our evaluations, Cosmos Policy achieves state-of-the-art performance on the LIBERO and RoboCasa simulation benchmarks (98.5% and 67.1% average success rates, respectively) and the highest average score in challenging real-world bimanual manipulation tasks, outperforming strong diffusion policies trained from scratch, video model-based policies, and state-of-the-art vision-language-action models fine-tuned on the same robot demonstrations. Furthermore, given policy rollout data, Cosmos Policy can learn from experience to refine its world model and value function and leverage model-based planning to achieve even higher success rates in challenging tasks. We release code, models, and training data at https://research.nvidia.com/labs/dir/cosmos-policy/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。