无需演示即可让机器人零样本执行新任务，靠生成视频驱动动作流
NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos
- 用视频生成模型合成任务视频，再提取3D物体运动流作为行动依据
- 在无示范情况下，成功实现刚体、关节和柔性物体的零样本操作
- 适配机械臂与四足机器人，跨平台迁移能力强，无需针对设备训练
让机器人实现零样本的新操作任务是机器人领域的核心目标。现有方法大多依赖分布内任务或需针对具体硬件微调数据，限制了跨平台迁移能力。我们提出NovaFlow，一种自主操作框架，能将任务描述直接转化为目标机器人的可执行计划，无需任何示范。给定任务描述后，NovaFlow利用视频生成模型合成视频，并通过现成感知模块将其提炼为3D可行动物体运动流。基于该运动流，对刚性物体计算相对位姿，通过抓取提议与轨迹优化生成机器人动作；对柔性物体，则以该流作为基于粒子的动力学模型规划的跟踪目标。通过解耦任务理解与底层控制，NovaFlow天然支持跨硬件迁移。我们在桌面Franka机械臂与Spot四足机器人上验证了刚体、关节及柔性物体的操作任务，均实现无需示范或设备特化训练的有效零样本执行。
原文摘要 · Abstract (English)
Enabling robots to execute novel manipulation tasks zero-shot is a central goal in robotics. Most existing methods assume in-distribution tasks or rely on fine-tuning with embodiment-matched data, limiting transfer across platforms. We present NovaFlow, an autonomous manipulation framework that converts a task description into an actionable plan for a target robot without any demonstrations. Given a task description, NovaFlow synthesizes a video using a video generation model and distills it into 3D actionable object flow using off-the-shelf perception modules. From the object flow, it computes relative poses for rigid objects and realizes them as robot actions via grasp proposals and trajectory optimization. For deformable objects, this flow serves as a tracking objective for model-based planning with a particle-based dynamics model. By decoupling task understanding from low-level control, NovaFlow naturally transfers across embodiments. We validate on rigid, articulated, and deformable object manipulation tasks using a table-top Franka arm and a Spot quadrupedal mobile robot, and achieve effective zero-shot execution without demonstrations or embodiment-specific training. Project website: https://novaflow.lhy.xyz/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。