arXiv:2607.26579cs.ROcs.CV2026-07被引 1

用接触点轨迹统一人类与机器人的操作条件,实现跨实体动作迁移。

ContactFlow: A video action conditioning that transfers across embodiments

  • 用3D接触点轨迹表示操作,摆脱具体机械结构限制。
  • 在DROID数据集和真实场景中验证了跨人机与机器人形态的迁移能力。
  • 适合做通用机器人规划与人机协作任务的研究者参考。

世界模型通过让智能体在执行前想象和验证动作后果,为机器人规划提供了前景。然而,当前基于视频的世界模型难以捕捉操纵中的物理约束,尤其是接触关系。此外,其动作条件常局限于特定实体(如平行夹爪)。我们提出「Contact Flow」——一种不依赖具体实体的动作表示方法,通过编码操作者与目标物体间3D接触点的运动轨迹来表征操纵过程。该方法摒弃了操作者外观与运动学特征,从而为人类示范和机器人执行提供统一的条件信号。因此,我们可基于人类与机器人交互视频,利用Contact Flow训练大规模视频生成模型,构建能预测物理上合理操纵结果的世界模型。我们将该模型集成到“提议-想象-验证-执行”流程中,由视觉语言模型评估生成的轨迹后再执行。在DROID数据集及真实桌面操纵任务上的实验表明,Contact Flow实现了从人类示范到不同机器人实体间的有效迁移。

原文摘要 · Abstract (English)

World models offer a promising route toward robot planning by enabling agents to imagine and verify the consequences of actions before execution. However, current video-based world models often struggle to capture the physical constraints that govern manipulation, particularly contact. Further, their action conditioning is often constrained to specific embodiments such as parallel grippers. We propose \emph{Contact Flow}, an embodiment-agnostic action representation that encodes manipulation through the trajectory of 3D contact points between an actor and a target object. By discarding actor-specific appearance and kinematics, Contact Flow provides a shared conditioning signal for both human demonstrations and robotic execution. Therefore, we can train a large-scale video generative model on both human and robotic object interaction videos conditioned on Contact Flow, yielding a world model that predicts physically plausible manipulation outcomes. We integrate this model into a propose-imagine-verify-act pipeline, where generated rollouts are assessed by a vision-language model before execution. Experiments on the DROID dataset and real-world tabletop manipulation tasks demonstrate that Contact Flow enables transfer between human demonstrations and different robotic embodiments.

世界模型动作迁移机器人操控视频生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。