arXiv:2512.24766cs.ROcs.AI2025-12被引 30

用3D物体运动流连接视频生成与机器人操控,实现零样本开放世界操作。

Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow

  • 以3D物体运动流为中间表示,连接视频生成与机器人控制
  • 无需任务特定示范,可操控刚体、柔性、颗粒等多类物体
  • 在仿真和真实场景中验证了方法的通用性与可扩展性

生成式视频建模已成为一种强大的工具,可用于零样本推理开放世界中的合理物理交互。然而,如何将人类主导的动作转化为机器人系统所需的底层指令仍具挑战。我们观察到,在给定初始图像和任务指令时,这类模型能出色合成合理的物体运动。为此,我们提出Dream2Flow框架,通过3D物体运动流作为中间表示,连接视频生成与机器人控制。该方法从生成视频中重建3D物体运动,并将操作建模为物体轨迹跟踪。通过分离状态变化与实现这些变化的执行器,Dream2Flow克服了具身差距,实现了预训练视频模型对多样化物体(包括刚体、铰接体、可变形体和颗粒体)的零样本引导。通过轨迹优化或强化学习,Dream2Flow将重建的3D物体运动流转换为可执行的底层指令,无需任务特定示范。仿真与真实世界实验表明,3D物体运动流是适应视频生成模型至开放世界机器人操作的通用且可扩展接口。视频与可视化见https://dream2flow.github.io/。

原文摘要 · Abstract (English)

Generative video modeling has emerged as a compelling tool to zero-shot reason about plausible physical interactions for open-world manipulation. Yet, it remains a challenge to translate such human-led motions into the low-level actions demanded by robotic systems. We observe that given an initial image and task instruction, these models excel at synthesizing sensible object motions. Thus, we introduce Dream2Flow, a framework that bridges video generation and robotic control through 3D object flow as an intermediate representation. Our method reconstructs 3D object motions from generated videos and formulates manipulation as object trajectory tracking. By separating the state changes from the actuators that realize those changes, Dream2Flow overcomes the embodiment gap and enables zero-shot guidance from pre-trained video models to manipulate objects of diverse categories-including rigid, articulated, deformable, and granular. Through trajectory optimization or reinforcement learning, Dream2Flow converts reconstructed 3D object flow into executable low-level commands without task-specific demonstrations. Simulation and real-world experiments highlight 3D object flow as a general and scalable interface for adapting video generation models to open-world robotic manipulation. Videos and visualizations are available at https://dream2flow.github.io/.

视频生成机器人操控3D运动流零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。