从视频中学习人类动作，精准定位手物接触点并生成机器人可执行的多样动作。
Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Open-World Contact Localization
- 以手为中心，通过交互线索推断手部3D运动与接触时间区间。
- 在多种任务上实现86%成功率，优于人工操作轨迹。
- 无需物体描述或状态追踪，适合真实场景下的机器人学习。
由于手物交互噪声、未见物体部分观测以及跨体态差异，从人类视频示范中学习仍具挑战。为此，我们提出 extit{HOWTransfer}（手-对象-开放世界转移），一种以手为中心的框架，将人类示范提炼为具有接触感知、分类信息且多样化的机器人轨迹。该方法不依赖物体特定描述、视觉语言查询或显式物体状态追踪，而是通过分析观察到的手物交互线索，恢复时序一致的3D手部运动并定位接触起始时刻。所得到的接触时间点被用于将人类抓取意图转换为多模态并行夹爪假设，并沿恢复的腕部轨迹传播，生成机器人可执行的动作。最后，通过轨迹编辑阶段优化接触对齐，从单一示范生成多样化可执行变体。在多种操作任务上的实验表明， extit{HOWTransfer}实现了高精度接触定位和高质量机器人动作重定向，成功率高达86%，在盲测偏好研究中优于人工操作轨迹。
原文摘要 · Abstract (English)
Learning from human video demonstrations remains challenging due to noisy hand-object interactions, unseen objects with partial observation, and cross-embodiment discrepancy. To address these challenges, we present \textit{HOWTransfer} (\emph{H}and-\emph{O}bject \emph{O}pen-\emph{W}orld Transfer), a hand-centric framework that distills human demonstrations into contact-aware, taxonomy-informed, and diverse robotic trajectories. Instead of relying on object-specific descriptions, vision-language queries, or explicit object-state tracking, \emph{HOWTransfer} recovers temporally consistent 3D hand motion and localizes temporal contact intervals by reasoning over observed hand-object interaction cues. The localized contact onsets are then used to retarget human grasp intent into multi-modal parallel-jaw grasp hypotheses, which are propagated along the recovered wrist trajectory to generate robot-executable motions. Finally, a trajectory editing stage refines contact alignment and produces diverse executable variants from a single demonstration. Experiments across diverse manipulation tasks show that \emph{HOWTransfer} enables accurate contact localization and high-quality robot motion retargeting with $86\%$ success, which is preferred over teleoperated trajectories in a blinded preference study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。