无需配对数据，将人类操作视频转为物理真实机器人动作视频
H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos
- 用图像修复+视觉提示替代配对数据训练生成模型
- 生成的机器人动作更符合物理规律，且时间连贯性好
- 适合希望低成本获取机器人操作数据的研究者
从日常人类-物体交互视频中学习操作技能的机器人，可在不进行繁琐机器人数据采集的情况下获得广泛能力。本文提出一种视频到视频的转换框架，将普通人类操作视频转化为运动一致、物理真实的机器人操作视频。该方法仅需未配对的机器人视频进行训练，无需任何成对的人类-机器人视频，便于规模化。我们引入可迁移的表示来弥合具身差异：在训练时，通过修复机器人手臂区域得到干净背景，并叠加简单视觉提示（标记和箭头指示夹爪位置与方向），以此条件生成模型将机器人手臂重新插入场景。测试时，对人类视频同样执行修复人物并叠加人体姿态提示，生成高质量模仿人类动作的机器人视频。我们采用上下文学习方式微调SOTA视频扩散模型（Wan 2.2），以保证时间连贯性并利用其丰富的先验知识。实验结果表明，本方法生成的机器人动作显著更真实、更具物理合理性，为从无标签人类视频中规模化学习机器人技能指明了新方向。
原文摘要 · Abstract (English)
Robots that learn manipulation skills from everyday human videos could acquire broad capabilities without tedious robot data collection. We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos with realistic, physically grounded interactions. Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale. We introduce a transferable representation that bridges the embodiment gap: by inpainting the robot arm in training videos to obtain a clean background and overlaying a simple visual cue (a marker and arrow indicating the gripper's position and orientation), we can condition a generative model to insert the robot arm back into the scene. At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions. We fine-tune a SOTA video diffusion model (Wan 2.2) in an in-context learning manner to ensure temporal coherence and leveraging of its rich prior knowledge. Empirical results demonstrate that our approach achieves significantly more realistic and grounded robot motions compared to baselines, pointing to a promising direction for scaling up robot learning from unlabeled human videos. Project page: https://showlab.github.io/H2R-Grounder/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。