无需任务训练,直接模仿生成视频实现机器人零样本物体交互。
GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training

- 通过生成视频并提取接触事件,构建物体中心几何约束。
- 在仿真中重构场景,优化参考轨迹以适应新物体姿态。
- 适合希望快速部署多类型交互任务的机器人研发者。
人形机器人与物体交互(HOI)是其核心能力,但动态平衡与稳定交互之间存在紧密耦合,难以实现。现有方法常需耗时的任务特定策略训练或依赖刚性轨迹重放,难以适应新场景。本文提出GenHOI框架,仅通过模仿单个生成视频即可实现零样本的多样化物体交互,无需任务特定训练或物理演示数据。首先在仿真中重建机器人-物体场景并渲染首帧图像,结合语言指令生成任务导向的交互视频。分析生成视频以识别交互相关的接触事件并估计手物接触区域,将其编码为物体中心的几何约束,将视觉交互线索转化为物理可执行的优化先验。基于这些先验,对视频中恢复的参考运动进行优化和光滑化,解决2D视频生成固有的尺度模糊问题,并使单一参考轨迹适配未见过的机器人-物体相对姿态。最终由闭环跟踪控制器执行优化轨迹。在大量仿真与真实实验中验证了该框架在盒体抓取、非对称双臂搬椅、从下方抬桌及圆柱物体包覆等多种交互任务上的有效性。
原文摘要 · Abstract (English)
Humanoid-Object Interaction (HOI) is a fundamental capability for humanoid robots, yet it remains challenging due to the tight coupling between dynamic balance and stable interaction with diverse objects. Existing methods often require time-consuming task-specific policy training or rely on rigid trajectory replay, which limits their ability to accommodate novel interaction scenarios. In this work, we present \textit{GenHOI}, a simple yet effective framework that enables humanoid robots to perform diverse object-interaction tasks in a zero-shot manner by directly imitating a single generated video, without task-specific training or physical demonstration data. GenHOI first reconstructs the robot-object scene in simulation and renders a first-frame image, which, together with the language command, conditions the synthesis of a task-oriented interaction video. The generated video is then analyzed to identify interaction-relevant contact events and estimate hand-object contact regions, which are encoded as object-centric geometric constraints that convert visual interaction cues into physically grounded optimization priors. Guided by these priors, the reference motion recovered from the video is refined and smoothed to resolve the scale ambiguity inherent in 2D video generation, while adapting a single reference trajectory to unseen robot-object relative poses. The optimized trajectory is finally executed by a closed-loop tracking controller. We validate the proposed framework in extensive simulation and real-world experiments across diverse object-interaction tasks, including box grasping, asymmetric bimanual chair carrying, table lifting from below, and cylindrical-object enveloping.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。