用单目摄像头生成机器人腕部视角，让机器人从人类示范视频学操作。
WARPED: Wrist-Aligned Rendering for Robot Policy Learning from Egocentric Human Demonstrations
- 通过视觉大模型初始化场景，追踪手物交互并重定向轨迹到机械臂。
- 用高斯点云渲染出逼真的腕部视角图像，直接训练机器人策略。
- 只需5-8倍少的数据采集时间，就能达到与真实操控相当的效果。
近期基于人类示范的学习在缓解训练鲁棒视觉运动策略所需数据收集的可扩展性和高成本问题上展现出良好前景。然而，现有方法通常依赖多视角摄像头、深度传感器或定制硬件，且多局限于第三人称或第一人称摄像头下的策略执行。本文提出WARPED框架，仅使用单目RGB数据，即可从人类示范视频中合成真实的腕部视角观测，以支持视觉运动策略的训练。系统首先利用视觉基础模型初始化交互场景，再通过手物交互追踪管线跟踪手部与被操作物体，并将轨迹重定向至机器人末端执行器。最后，采用高斯点云（Gaussian Splatting）生成逼真的腕部视角图像，直接用于机器人策略训练。实验表明，在五个桌面操作任务上，WARPED达到与使用遥操作示范数据训练的策略相媲美的成功率，同时数据采集时间减少5-8倍。
原文摘要 · Abstract (English)
Recent advancements in learning from human demonstration have shown promising results in addressing the scalability and high cost of data collection required to train robust visuomotor policies. However, existing approaches are often constrained by a reliance on multiview camera setups, depth sensors, or custom hardware and are typically limited to policy execution from third-person or egocentric cameras. In this paper, we present WARPED, a framework designed to synthesize realistic wrist-view observations from human demonstration videos to facilitate the training of visuomotor policies using only monocular RGB data. With data collected from an egocentric RGB camera, our system leverages vision foundation models to initialize the interactive scene. A hand-object interaction pipeline is then employed to track the hand and manipulated object and retarget the trajectories to a robotic end-effector. Lastly, photo-realistic wrist-view observations are synthesized via Gaussian Splatting to directly train a robotic policy. We demonstrate that WARPED achieves success rates comparable to policies trained on teleoperated demonstration data for five tabletop manipulation tasks, while requiring 5-8x less data collection time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。